r47 | ozzloy | 2007-09-13 12:54:13 -0700 (Thu, 13 Sep 2007) | 1 line
------------------------------------------------------------------------
r46 | ozzloy | 2007-09-13 12:44:41 -0700 (Thu, 13 Sep 2007) | 1 line
------------------------------------------------------------------------
r45 | ozzloy | 2007-09-13 12:40:11 -0700 (Thu, 13 Sep 2007) | 1 line
adding fisher stuff
i have had some fisherface code lying around for a while. originally it was written with the goal of completing cse152 homework assignments. i reworked the code so that it will work with the picElems i use to represent the images and their metadata.
Wednesday, September 26, 2007
r42 r43
r43 | ozzloy | 2007-09-12 22:57:16 -0700 (Wed, 12 Sep 2007) | 2 lines
renaming continues for distinguishing between eigen and fisher
------------------------------------------------------------------------
r42 | ozzloy | 2007-09-12 22:49:20 -0700 (Wed, 12 Sep 2007) | 1 line
prepending 'eigen' to filenames to distinguish from fisher
as a result of moving all the functions around and keeping them all in the same directory, i have to rename functions (and therefore the files to which they belong) to make them distinct. of course this means i have to update everywhere the function is called as well. i don't really have a good way of doing this other than using "grep" and looking at each instance to make sure it should be updated to the new name. another possibility is that the function name happens to appear in a comment and should not be updated. sed and awk could probably do this for me more quickly. maybe i will teach myself how to use them.
renaming continues for distinguishing between eigen and fisher
------------------------------------------------------------------------
r42 | ozzloy | 2007-09-12 22:49:20 -0700 (Wed, 12 Sep 2007) | 1 line
prepending 'eigen' to filenames to distinguish from fisher
as a result of moving all the functions around and keeping them all in the same directory, i have to rename functions (and therefore the files to which they belong) to make them distinct. of course this means i have to update everywhere the function is called as well. i don't really have a good way of doing this other than using "grep" and looking at each instance to make sure it should be updated to the new name. another possibility is that the function name happens to appear in a comment and should not be updated. sed and awk could probably do this for me more quickly. maybe i will teach myself how to use them.
r41 too many open files
r41 | ozzloy | 2007-09-12 22:36:44 -0700 (Wed, 12 Sep 2007) | 3 lines
currently works for a small number of plots before dying.
oh i hate matlab again. there is some sort of "too many open files" error that is keeping me from using the picture grabbing function i just wrote. i worked out a couple different solutions to this:
the first is to open 1 out of 10 files that i was opening before. this actually made things go quite quickly. unfortunately it wasn't a real solution. the open file count would still max out and cause matlab to crash. at this point i would still have to restart matlab and re-initialize many variables.
i called mathworks and left a message detailing my problem. the next day i got a call back. we discussed the problem and he told me about a way to close all files. i put "fclose('all');" into the place of my code just after where i might have open files. this worked, but it also leads me to believe that the problem is not in my code, but in one of the functions i call, "imread(image_location)". i told the mathworks guy about this and he said he'd look into it. this 'solution' closes all files. that's not exactly what i want to do. right now it seems to be ok. however i can imagine a situation in which i would call this code and it would close all files, causing problems because i can't figure out why my files are closing. basically it's a problem for future users of this code.
currently works for a small number of plots before dying.
oh i hate matlab again. there is some sort of "too many open files" error that is keeping me from using the picture grabbing function i just wrote. i worked out a couple different solutions to this:
the first is to open 1 out of 10 files that i was opening before. this actually made things go quite quickly. unfortunately it wasn't a real solution. the open file count would still max out and cause matlab to crash. at this point i would still have to restart matlab and re-initialize many variables.
i called mathworks and left a message detailing my problem. the next day i got a call back. we discussed the problem and he told me about a way to close all files. i put "fclose('all');" into the place of my code just after where i might have open files. this worked, but it also leads me to believe that the problem is not in my code, but in one of the functions i call, "imread(image_location)". i told the mathworks guy about this and he said he'd look into it. this 'solution' closes all files. that's not exactly what i want to do. right now it seems to be ok. however i can imagine a situation in which i would call this code and it would close all files, causing problems because i can't figure out why my files are closing. basically it's a problem for future users of this code.
r39 database like access to pictures
r39 | ozzloy | 2007-09-12 15:47:23 -0700 (Wed, 12 Sep 2007) | 3 lines
now i can generate a list of complete paths to pictures, saved in a single
variable
i now have a single function that does all the directory traversals. this function takes place of the other find*.m functions. this makes the perl functions obsolete. still, it was fun to learn perl. it acts kinda like a query-able database from the file system. i'm sure there are better ways of doing this, but right now it's a bunch of nested for-loops that iterate over the different possible paths. if no path is specified, it will search for valid paths under the current working directory and make that the new list to iterate over.
now i can generate a list of complete paths to pictures, saved in a single
variable
i now have a single function that does all the directory traversals. this function takes place of the other find*.m functions. this makes the perl functions obsolete. still, it was fun to learn perl. it acts kinda like a query-able database from the file system. i'm sure there are better ways of doing this, but right now it's a bunch of nested for-loops that iterate over the different possible paths. if no path is specified, it will search for valid paths under the current working directory and make that the new list to iterate over.
r36
r36 | ozzloy | 2007-09-12 14:01:09 -0700 (Wed, 12 Sep 2007) | 3 lines
adding a few more matlab functions so i can start deleting stuff without
worrying
i have created a bunch of matlab scripts that i haven't checked into the repository. when i start writing them, i'm not sure what i will want to keep and put into the repository for me and serge. i have decided that i can just add them and later remove the ones i do not want. still, it would be nice to be able to use a distributed VCS. then i could keep track of those things locally and only put them up for other people when i was certain i wanted to.
adding a few more matlab functions so i can start deleting stuff without
worrying
i have created a bunch of matlab scripts that i haven't checked into the repository. when i start writing them, i'm not sure what i will want to keep and put into the repository for me and serge. i have decided that i can just add them and later remove the ones i do not want. still, it would be nice to be able to use a distributed VCS. then i could keep track of those things locally and only put them up for other people when i was certain i wanted to.
r35 initial directory walking in matlab
r35 | ozzloy | 2007-09-12 13:52:43 -0700 (Wed, 12 Sep 2007) | 3 lines
directory walking is now possible entirely in matlab using find*.m
i have completely implemented the ability to search out and grab images off the disk. i have a bunch of find*.m files (egs: "findPPMs.m"). this is better than using the perl in a different directory because i don't have to hardcode the location of the perl scripts. however it does clutter up the current directory with a bunch of find*.m files, which is annoying. also, this makes the work a bit more portable. all i need to do is install matalb on another machine and it should be able to run these scripts.
directory walking is now possible entirely in matlab using find*.m
i have completely implemented the ability to search out and grab images off the disk. i have a bunch of find*.m files (egs: "findPPMs.m"). this is better than using the perl in a different directory because i don't have to hardcode the location of the perl scripts. however it does clutter up the current directory with a bunch of find*.m files, which is annoying. also, this makes the work a bit more portable. all i need to do is install matalb on another machine and it should be able to run these scripts.
r34
r34 | ozzloy | 2007-09-04 11:23:13 -0700 (Tue, 04 Sep 2007) | 5 lines
adding the fresh directory. need to delete all *.png
accidentally removed all fresh/out
i'm a doofus
matlab is such a pain. i have not been able to separate functions into packages, and every function must be in its own file. the result is that i have many many files in the current working directory that do many distinct things dealing with distinct concepts. in the same directory i have files that will read images off the disk and files that will calculate the top eigenfaces using fisher's linear discriminant. the only thing i have in a separate directory is the directory traversal which is written in perl and whose locations are hard-coded in the matlab functions. i need to learn to deal with matlab paths. maybe i could have one uber-function in the top directory that loads the paths for everything.
adding the fresh directory. need to delete all *.png
accidentally removed all fresh/out
i'm a doofus
matlab is such a pain. i have not been able to separate functions into packages, and every function must be in its own file. the result is that i have many many files in the current working directory that do many distinct things dealing with distinct concepts. in the same directory i have files that will read images off the disk and files that will calculate the top eigenfaces using fisher's linear discriminant. the only thing i have in a separate directory is the directory traversal which is written in perl and whose locations are hard-coded in the matlab functions. i need to learn to deal with matlab paths. maybe i could have one uber-function in the top directory that loads the paths for everything.
r33
r33 | ozzloy | 2007-08-10 13:30:53 -0700 (Fri, 10 Aug 2007) | 9 lines
graphTop9Eigs now scales the top 9 eigenvectors such that the lowest value on
any pixel+color element is 0, and the highest is 255. the rest are truncated to
integral values somewhere in between
createPicElem is in progress. it will hold all the information about an image,
such as the pixel+channel column vector, the number of rows, cols, and which
colors, the set ppm, the picture's ppm. NOTE: createPicElem doesn't actually do
this stuff yet.
my work is mostly mathematical. most of the time i don't need to see an actual picture, even though i'm dealing with thousands of them all the time. but i do need to represent what i'm doing for other people to gauge my progress. for the people only loosely familiar with the details, i must represent what i am doing in a way that glosses over details and emphasizes important parts.
top9eigs will take the top 9 eigenvalues and turn them into pictures. this function was not maintained very well. it was not tested for robustness to different inputs. it did not handle all correct inputs correctly. now it handles a wider variety of inputs and scales them correctly. i use this to generate a graphical representation of the mathematics performed. i am then able to use this as a visual aid as i explain to investors what progress i have made.
i must also create a way to store meta data about the pictures. i have decided to do this with a struct. i want to spend the least amount of time possible building this, so i'm not going to learn anything new first. it seems like using structs would be the quickest way to do this. i might do objects later.
graphTop9Eigs now scales the top 9 eigenvectors such that the lowest value on
any pixel+color element is 0, and the highest is 255. the rest are truncated to
integral values somewhere in between
createPicElem is in progress. it will hold all the information about an image,
such as the pixel+channel column vector, the number of rows, cols, and which
colors, the set ppm, the picture's ppm. NOTE: createPicElem doesn't actually do
this stuff yet.
my work is mostly mathematical. most of the time i don't need to see an actual picture, even though i'm dealing with thousands of them all the time. but i do need to represent what i'm doing for other people to gauge my progress. for the people only loosely familiar with the details, i must represent what i am doing in a way that glosses over details and emphasizes important parts.
top9eigs will take the top 9 eigenvalues and turn them into pictures. this function was not maintained very well. it was not tested for robustness to different inputs. it did not handle all correct inputs correctly. now it handles a wider variety of inputs and scales them correctly. i use this to generate a graphical representation of the mathematics performed. i am then able to use this as a visual aid as i explain to investors what progress i have made.
i must also create a way to store meta data about the pictures. i have decided to do this with a struct. i want to spend the least amount of time possible building this, so i'm not going to learn anything new first. it seems like using structs would be the quickest way to do this. i might do objects later.
Tuesday, September 25, 2007
r30
r30 | ozzloy | 2007-08-06 17:01:10 -0700 (Mon, 06 Aug 2007) | 4 lines
starting them fisherFaces
:wq
fisher faces is only slightly more complicated than pca from an overview perspective. but the details. oh man, the details!
first, you have to keep track of the class information. this means you have to associate with every point of data the class to which it belongs. i have had trouble doing this in matlab because i don't know how to deal with strings very well. originally i had most of the string handling done in perl scripts in a separate directory. dealing with paths and separate directories in matlab is a pain though. eventually i got the hang of cell arrays and strings in matlab. i still don't really understand it quite well enough to quiz someone else on it or write code confidently. instead i must write some code, see if it works, change the syntax, see what the result is. this process is very time consuming. especially because it keeps you from simply writing code to express your thoughts. instead your thoughts are interrupted while you think about getting the first thoughts syntactically correct. you shouldn't have to fit your thoughts into the representation provided by matlab, matlab should be able to represent your thoughts directly. hooray object oriented. boo matlab's implementation.
starting them fisherFaces
:wq
fisher faces is only slightly more complicated than pca from an overview perspective. but the details. oh man, the details!
first, you have to keep track of the class information. this means you have to associate with every point of data the class to which it belongs. i have had trouble doing this in matlab because i don't know how to deal with strings very well. originally i had most of the string handling done in perl scripts in a separate directory. dealing with paths and separate directories in matlab is a pain though. eventually i got the hang of cell arrays and strings in matlab. i still don't really understand it quite well enough to quiz someone else on it or write code confidently. instead i must write some code, see if it works, change the syntax, see what the result is. this process is very time consuming. especially because it keeps you from simply writing code to express your thoughts. instead your thoughts are interrupted while you think about getting the first thoughts syntactically correct. you shouldn't have to fit your thoughts into the representation provided by matlab, matlab should be able to represent your thoughts directly. hooray object oriented. boo matlab's implementation.
r25
r25 | ozzloy | 2007-08-06 14:18:13 -0700 (Mon, 06 Aug 2007) | 4 lines
adding getSetDimension. need to update get[Faces[ChromLum]] to not duplicate
functionality.
lalala
i have found that writing these scripts tends to result in duplicated code. i also am having trouble with how i should grab the pictures off the disc. i have functions that begin with "get" and allow me to grab based on different criteria. Chrom and Lum were an idea from Serge Belongie to rearrange the red green and blue channels into 1 channel of luminance and 2 channels of chroma information. using this transform, we found that we could use 2 channel's worth of information instead of 3 with nearly the same accuracy.
i want to switch my versioning from svn to git or something else distributed like it. i can't though because serge might have problems with it. i'd like to keep track of the changes i am making locally and only when i feel they are good would i push them to the server for serge.
adding getSetDimension. need to update get[Faces[ChromLum]] to not duplicate
functionality.
lalala
i have found that writing these scripts tends to result in duplicated code. i also am having trouble with how i should grab the pictures off the disc. i have functions that begin with "get" and allow me to grab based on different criteria. Chrom and Lum were an idea from Serge Belongie to rearrange the red green and blue channels into 1 channel of luminance and 2 channels of chroma information. using this transform, we found that we could use 2 channel's worth of information instead of 3 with nearly the same accuracy.
i want to switch my versioning from svn to git or something else distributed like it. i can't though because serge might have problems with it. i'd like to keep track of the changes i am making locally and only when i feel they are good would i push them to the server for serge.
r24
r24 | ozzloy | 2007-08-06 13:41:05 -0700 (Mon, 06 Aug 2007) | 6 lines
made asdf.m a function that calls avaak0.m. i should rewrite so that i can save
all the things that take a while to compute.
that way they can be used with minimal scripting
need to create a [rows,cols,colors] = getDimensions(fileSet);
after creating a bunch of separate functions that calculated steps along the way to getting eigenvectors, i strung them together in abstracted functions. the abstracted functions would hide all the intermediate changes. the functions took a long time to run. i want to have a way of running just a small portion of the full process. this can be done by having everything run as a script instead of a function. i started to do this with asdf and avaak0, but i generally find that i want to turn those into functions to which i can pass parameters. it's quite a handy way of developing.
made asdf.m a function that calls avaak0.m. i should rewrite so that i can save
all the things that take a while to compute.
that way they can be used with minimal scripting
need to create a [rows,cols,colors] = getDimensions(fileSet);
after creating a bunch of separate functions that calculated steps along the way to getting eigenvectors, i strung them together in abstracted functions. the abstracted functions would hide all the intermediate changes. the functions took a long time to run. i want to have a way of running just a small portion of the full process. this can be done by having everything run as a script instead of a function. i started to do this with asdf and avaak0, but i generally find that i want to turn those into functions to which i can pass parameters. it's quite a handy way of developing.
r12 through r21
because the initial code was written for cse 152, i have had to clean up the directory structure and move functions around to their own files. matlab doesn't easily allow functions in one file to be called from another file. i actually wish i could just use java for this.
i also had to make some changes to accommodate windows. i am working with serge. he uses windows and mac. i might be able to do symlinks on windows, but i don't already know how. to be sure that it will work on his computer, i just made up a folder with repeated information. if only there were some way the file system were accessible like database instead of a hierarchy... anyways, it's difficult to make sure that this code will work on someone else's system.
i also had to make some changes to accommodate windows. i am working with serge. he uses windows and mac. i might be able to do symlinks on windows, but i don't already know how. to be sure that it will work on his computer, i just made up a folder with repeated information. if only there were some way the file system were accessible like database instead of a hierarchy... anyways, it's difficult to make sure that this code will work on someone else's system.
r6
r6 | ozzloy | 2007-07-17 10:24:39 -0700 (Tue, 17 Jul 2007) | 5 lines
GOLD. in matlab, run 'h' and you'll get::
pictures of top 9 eigenvectors
plot of top 9 response curves and dosing curve
plot of top eigenVector response curve and dosing curve
i have created the plots of the top 9 eigen vectors. the eigenvectors are repackaged from points in the high dimensional space to pictures in 2d.
this work was based mostly on the work i did in cse 152 for computer vision. in this class i started learning how to use matlab. i created many scripts for that were divided based on which part of the homework they solved. as a result the scripts were not well organized for doing the job of creating eigenvectors generically. i had to update the structure of the scripts to separate out responsibilities. eventually the code from 152 was rewritten. it now handles colored pictures of many sizes. the size of the pictures is limited by the computation time.
GOLD. in matlab, run 'h' and you'll get::
pictures of top 9 eigenvectors
plot of top 9 response curves and dosing curve
plot of top eigenVector response curve and dosing curve
i have created the plots of the top 9 eigen vectors. the eigenvectors are repackaged from points in the high dimensional space to pictures in 2d.
this work was based mostly on the work i did in cse 152 for computer vision. in this class i started learning how to use matlab. i created many scripts for that were divided based on which part of the homework they solved. as a result the scripts were not well organized for doing the job of creating eigenvectors generically. i had to update the structure of the scripts to separate out responsibilities. eventually the code from 152 was rewritten. it now handles colored pictures of many sizes. the size of the pictures is limited by the computation time.
back log begins:
r3 | ozzloy | 2007-07-16 16:26:08 -0700 (Mon, 16 Jul 2007) | 3 lines
creating a subdir for eigenFaces
The project's ultimate goal is to make a chemical sensor. That's useful for bomb detection. but this has been done, so why do it again? traditional chemical sensors are one-time use. a paper strip exposed to a particular agent will turn from the color it is to some other color. the paper actively changes and does not return to its original state upon removal of the agent. the sensors developed by mike sailor called rugate filters passively change color. they will change color as a result of changes in the environment. they will react to many agents. the rugate filters will yield the same color pattern when exposed to the same environment. this means they can repeatedly detect increased levels of a toxin. we do not have to discard rugate filters after one use. as a result, rugate filters can potentially be continuously monitoring the environment for changes in chemical levels. this requires that the filter be monitored for long amounts of time. a low cost camera provided by avaak can wirelessly report its findings back to a central computer capable of interpreting the changes in the image and determining the chemicals present in the environment.
my project involves collecting pictures of these sensors reacting to known concentrations of chemicals. each picture is a data point. i must categorize the changes. after quantifying the changes from picture to picture, i must determine if the change is correlated with the change in the chemicals of the environment. to determine where the changes in the pictures are the largest, i used principal component analysis, PCA. the cross between PCA and image analysis is called "eigen faces" because it was used with pictures of faces and yielded a high degree of success.
at this point i have collected a bunch of pictures and must start finding out where they change the most for a given change in the chemical environment. typically this change will be from 0 ppm to 50k ppm of one analyte.
creating a subdir for eigenFaces
The project's ultimate goal is to make a chemical sensor. That's useful for bomb detection. but this has been done, so why do it again? traditional chemical sensors are one-time use. a paper strip exposed to a particular agent will turn from the color it is to some other color. the paper actively changes and does not return to its original state upon removal of the agent. the sensors developed by mike sailor called rugate filters passively change color. they will change color as a result of changes in the environment. they will react to many agents. the rugate filters will yield the same color pattern when exposed to the same environment. this means they can repeatedly detect increased levels of a toxin. we do not have to discard rugate filters after one use. as a result, rugate filters can potentially be continuously monitoring the environment for changes in chemical levels. this requires that the filter be monitored for long amounts of time. a low cost camera provided by avaak can wirelessly report its findings back to a central computer capable of interpreting the changes in the image and determining the chemicals present in the environment.
my project involves collecting pictures of these sensors reacting to known concentrations of chemicals. each picture is a data point. i must categorize the changes. after quantifying the changes from picture to picture, i must determine if the change is correlated with the change in the chemicals of the environment. to determine where the changes in the pictures are the largest, i used principal component analysis, PCA. the cross between PCA and image analysis is called "eigen faces" because it was used with pictures of faces and yielded a high degree of success.
at this point i have collected a bunch of pictures and must start finding out where they change the most for a given change in the chemical environment. typically this change will be from 0 ppm to 50k ppm of one analyte.
writing some journal entries
turns out i don't have to do this, it'll just help when i'm writing my bs reflective essay.
so my next entry and those after will be going over svn logs and my calendar while keeping the practicum reflective questions in mind.
what is the point of this class. through my life, when am i going to use the knowledge i gained from this class? at what point am i going to do something different than i would have done without this class, and why is it better than what i would have done? this class' lack of definition causes me a large amount of frustration.
so my next entry and those after will be going over svn logs and my calendar while keeping the practicum reflective questions in mind.
what is the point of this class. through my life, when am i going to use the knowledge i gained from this class? at what point am i going to do something different than i would have done without this class, and why is it better than what i would have done? this class' lack of definition causes me a large amount of frustration.
Thursday, September 20, 2007
finished with flda (kinda)
i got the framework for flda working. but the resulting graphs look like noise mostly. i have sent an email to serge asking him to review my math. matlab needs to have better support for object oriented programming. everything i'm coding right now is in its own file and if i want to create classes of my own, i'd have to create a new folder for each class and add that folder to the default path. this isn't hard, but it disrupts the flow of coding. while coding fisher's linear discriminant, it would have been quite useful to create an object that held the different classifications, and each classification could hold the elements belonging to that class. currently, this is all coded up using structs and arrays.
unfortunately, serge is out of the office. in the meantime i should get nearest neighbor and k-nearest neighbor working. i can use pca to generate test results.
unfortunately, serge is out of the office. in the meantime i should get nearest neighbor and k-nearest neighbor working. i can use pca to generate test results.
Thursday, September 13, 2007
i have just completed getElems();. here's what a typical path to an image looks like:
i just thought of something. i might be able to fake function pointers using some sort of eval function that converts strings into matlab commands. this would be such a horrible hack. i think first i should try to get some fisherface analysis. right now i don't have any of the ones i worked on weeks ago. since then i have spent no time on this. most of it has gone to my operating systems class instead.
../in/2007_08_20/AG020902/Heptane/100/00.22.39.jpgand here are some example getElem function calls:
e = getElems({'2007_08_20'},{},{'Heptane'},{'100'})when this is done, e holds all the pictures matching 2007_08_20/*/Heptane/100/*. and they will be ordered by time taken. sometimes the pictures in a directory will run across midnight, so some filenames will look like "12.59.59.jpg 00.00.01.jpg". this makes ordering by filename wrong. fortunately it's not hard to detect this. i calculate the time difference between the "latest" and "earliest" picture. if this time is too large, i know that the lower numbers (ones like "00.00.01.jpg") are actually 24 hrs later.
i just thought of something. i might be able to fake function pointers using some sort of eval function that converts strings into matlab commands. this would be such a horrible hack. i think first i should try to get some fisherface analysis. right now i don't have any of the ones i worked on weeks ago. since then i have spent no time on this. most of it has gone to my operating systems class instead.
too many open files
so i've just been dealing with too many open file descriptors in matlab. i called tech support and they told me that they don't normally deal with customers who have just a student license. i can't see the source of matlab's imread() function, so i don't know for sure that they aren't leaving a file open in there somewhere. the code that generates this error looks like this:
i went through several temporary fixes for this problem:
for ii = 1 : size( picPaths, 2);picElem( path) will create a picElem object with useful fields (such as rows, and cols). to do so, it uses imread() which must open the file. picPaths holds thousands of paths by the time this part of the code is reached. i can't narrow down exactly where files are being opened. the code above it has a bunch of calls to dir() which might be leaving files open. unfortunately i don't know how to get a list of currently open files, so it's hard to narrow down where files are being left open. maybe i could do something with lsof on the command line ... that doesn't sound too appetizing.
picElems(ii) = picElem(picPaths{ii});
end;
i went through several temporary fixes for this problem:
- my first fix was to limit the number of files actually opened. i called picElem 1 out of 10 times reached it. this fix allowed me to run the function more times and generate less detailed pictures, with 1/10th the number of data points.
- the second fix was one i got from the tech support guy. i can call fclose('all') to close all open files. calling this just after calling picElem, or just before the loop works. it allows me to generate full detailed graphs. it does close every open file though. i can imagine myself wanting to keep some files open at some point. for instance, the diary() file.
Subscribe to:
Posts (Atom)