Friday, September 13, 2013
Wednesday, July 31, 2013
Linux util 5
Whew!! My 5th linux util post...
1) Assuming you have SLURM basics, (well, I am just a beginner, so posting this very simple solution for the problem I faced), to force salloc to release the job allocation, find the process of that command using
b) kill -9 $pid
Or, If you have both salloc and srun in a single command,
c) kill -s HUP $pid
To see the signal identification for all the signals, use
man 7 signal
Just 'man signal' will take you to the C programming API for signaling.
2) Here is a simple way to find the number of words in each line of a file using awk:
awk '$0="line"NR": "NF' filename
Guess I got this tip again from SO, but this solution was not the accepted one.
3) If you pick only few lines output from make command to selectively fix the errors or warnings, (assuming you are in bash shell):
make -f Makefile clean; make -f Makefile 2 > &1 |grep 'error'
4) To identify the shell you are working on currently:
ps -p $$
From what I understand from the man pages, the -p option looks for pid list and $$ get the first process which is the shell itself. This prints the pid of the shell, terminal id etc. If you would rather want a concise output, then type:
echo $0
This was important to me change between bash and tcsh frequently when working on HPC machines of PNNL. The default login shell was tcsh, but I had do compilation and execution on bash. Most importantly, echo $SHELL did not help.
1) Assuming you have SLURM basics, (well, I am just a beginner, so posting this very simple solution for the problem I faced), to force salloc to release the job allocation, find the process of that command using
a) ps ax|grep $your_name|grep salloc
If you have reserved resources by separate salloc command (without invoking the job using srun), thenb) kill -9 $pid
Or, If you have both salloc and srun in a single command,
c) kill -s HUP $pid
To see the signal identification for all the signals, use
man 7 signal
Just 'man signal' will take you to the C programming API for signaling.
2) Here is a simple way to find the number of words in each line of a file using awk:
awk '$0="line"NR": "NF' filename
Guess I got this tip again from SO, but this solution was not the accepted one.
3) If you pick only few lines output from make command to selectively fix the errors or warnings, (assuming you are in bash shell):
make -f Makefile clean; make -f Makefile 2 > &1 |grep 'error'
4) To identify the shell you are working on currently:
ps -p $$
From what I understand from the man pages, the -p option looks for pid list and $$ get the first process which is the shell itself. This prints the pid of the shell, terminal id etc. If you would rather want a concise output, then type:
echo $0
This was important to me change between bash and tcsh frequently when working on HPC machines of PNNL. The default login shell was tcsh, but I had do compilation and execution on bash. Most importantly, echo $SHELL did not help.
Location:
Richland, WA, USA
Friday, June 14, 2013
Linux Util 4
Hmm, I had to open this new post because I cannot have more than 20 tags for a post.
1) June 14, 2013: Unresponsive system administrator, connection speed is slow and I am sitting in my PNNL intern office, not knowing how to see pdf files created using pdflatex in Natty - this is for my first ever paper from my internship.
Home directory based xpdf installation in Natty failed because of some missing software or dependencies - only admin could fix this. After losing hope here, today I tried my luck with evince, a medium-weight software, but definitely lighter than acroread, in Google for solutions to
"cannot parse arguments, cannot open display" problem.
And Bingo, thanks to StackExchange, all I had to do was to copy the evince binary from global directory to my home bin directory and set the alias for evince in bashrc. Phew!!! Wish I had known this before. Long live user Gilles!! He answered his own question, incidentally. A proper fix for this could be done only by the admin, apparently and well, then in my case, I have to wait indefinitely for this.
My months long wait is finally over :). Is it not good that we have something called home directory in linux?
2) About 8 years back (around 2006, 2007, way long back, huh), firefox used to save the bookmarks automatically to a file called bookmarks.html, that I used to just copy to the desired location, all in the command line itself. Did not realize this has changed with the recent versions. Now, I am looking for one url (Aeolus/ganglia, specifically) available in my office natty to my PNNL box, but I find my bookmarks.html file (at ~/.mozilla/firefox/profilename/) is actually way smaller than the number of bookmarks I find in my browser. Thanks to MozillaZine, I understood that the default behavior is not to save the bookmarks and that you have to edit the config file to force firefox to do that. Here is what I did:
1) June 14, 2013: Unresponsive system administrator, connection speed is slow and I am sitting in my PNNL intern office, not knowing how to see pdf files created using pdflatex in Natty - this is for my first ever paper from my internship.
Home directory based xpdf installation in Natty failed because of some missing software or dependencies - only admin could fix this. After losing hope here, today I tried my luck with evince, a medium-weight software, but definitely lighter than acroread, in Google for solutions to
"cannot parse arguments, cannot open display" problem.
And Bingo, thanks to StackExchange, all I had to do was to copy the evince binary from global directory to my home bin directory and set the alias for evince in bashrc. Phew!!! Wish I had known this before. Long live user Gilles!! He answered his own question, incidentally. A proper fix for this could be done only by the admin, apparently and well, then in my case, I have to wait indefinitely for this.
My months long wait is finally over :). Is it not good that we have something called home directory in linux?
2) About 8 years back (around 2006, 2007, way long back, huh), firefox used to save the bookmarks automatically to a file called bookmarks.html, that I used to just copy to the desired location, all in the command line itself. Did not realize this has changed with the recent versions. Now, I am looking for one url (Aeolus/ganglia, specifically) available in my office natty to my PNNL box, but I find my bookmarks.html file (at ~/.mozilla/firefox/profilename/) is actually way smaller than the number of bookmarks I find in my browser. Thanks to MozillaZine, I understood that the default behavior is not to save the bookmarks and that you have to edit the config file to force firefox to do that. Here is what I did:
- find the prefs.js file in .mozilla directory (should be available in the same location as your other profile specific files). The file header says you are not to edit the file, but I tried and it worked :)
- type the following as the last line of that file:
- user_pref("browser.bookmarks.autoExportHTML", true);
- save it. Reopen firefox and close it. This will update the local files.
- Now, open the bookmarks.html file and you will find all your saved bookmarks.
- Though you might have typed the 'user_pref' line towards the end, after the open-close of firefox, you will find that line sitting in a different place.
- Please follow the instructions at the header of the file prefs.js and the mozilla's about:config page, if you are not comfortable editing the .js file directly.
- Open the image in gimp and note the pixel information of the top, left and bottom, right corners. (Just hover your mouse to these places and you will see the pixel information in the bottom left corner in gimp changing.) Let us say you got, x1, y1 for top left and x2, y2 for bottom right.
- Calculate the width and height of the image from this information (subtract the smaller left value with larger left value and right with right). x2-x1 is the width, y2-y1 is the height.
- mogrify -crop widthxheight+x1+y1 imagename.png
Note the letter 'x' between the width and height values, it is not '*', the asterisk. Warning: the original image will be modified. Image quality was not modified at all.
Labels:
acroread,
apparmor,
bookmark,
config,
crop,
display,
evince,
export,
firefox,
gimp,
image,
mogrify,
pdfviewer,
preference,
stackexchange,
Xauthority,
xpdf
Location:
richland, WA, United States
Wednesday, January 16, 2013
Linux Util 3
1) To kill all processes of a process tree, say a bunch of child processes spawned by a parent process (note that all the child processes have pids greater than that of parent's):
kill -9 -(ppid)
where ppid stands for parent pid. (just a negation symbol before the parent process id)
Another way to find the parent id (is same as process group id):
stackoverflow-page helped to solve this problem.
2) If you work by logging to your office machine (Linux server) from windows client machine using NoMachine (NX), and when you open lot of xterms, it is oftentimes confusing what each xterm window corresponding based just on the title of it. Just follow these steps to get new informative title (but this is active only for this session of your login)
unset PROMPT_COMMAND (assuming you have this env variable set in your .bashrc)
echo -ne "\033]0;title\007"
It is important to blindly follow the characters and their sequence in the echo command. Obviously, they mean something, but to keep the brevity, I am refraining from giving the explanation here.
3) If you want to do copy/paste operation in xterm,
a) highlight the text using mouse
b) use middle button/scroll wheel of mouse to paste or
c) shift+insert to paste or when there is no middle button,or
d) simultaneously click both mouse buttons to emulate middle-button click
Thanks to Ubuntu Forums for this help.
4) All along I have been using pdftops - psselect - ps2pdf combination to select pages from a pdf document, but Today (Jan 23, 2013) I learnt about pdftk from
LinuxJournal to select pages from pdf document directly without converting to ps first. How nice!!
pdftk A=100p-inputfile.pdf cat A22-36 output outfile_p22-p36.pdf
Input file is 100p-inputfile.pdf. Though the name sounds confusing, there is no need to prefix the name with 100p (100 pages). It is just for this particular example. This command selects pages from 22 to 36 and creates an output file with just those pages.
5) April 17, 2013: Found this tip while wanting to put all docx files of submitted student homeworks in one directory to upload to google drive and open them there in order not to miss pictures or other openoffice incompatible things.
Thanks to Nixcraft I could do this and this is the first time I really found a way to use xargs command.
find . -name "*.docx" -print0 | xargs -0 -I file mv file onlydocs/
6) To sync a local directory to a remote directory:
You will be prompted to enter the password to the remote machine. If all else is correct, this should sync the files. Thanks to Nixcraft for this tip.
kill -9 -(ppid)
where ppid stands for parent pid. (just a negation symbol before the parent process id)
Another way to find the parent id (is same as process group id):
ps -eo "%p %r %c %a" stackoverflow-page helped to solve this problem.
2) If you work by logging to your office machine (Linux server) from windows client machine using NoMachine (NX), and when you open lot of xterms, it is oftentimes confusing what each xterm window corresponding based just on the title of it. Just follow these steps to get new informative title (but this is active only for this session of your login)
unset PROMPT_COMMAND (assuming you have this env variable set in your .bashrc)
echo -ne "\033]0;title\007"
It is important to blindly follow the characters and their sequence in the echo command. Obviously, they mean something, but to keep the brevity, I am refraining from giving the explanation here.
3) If you want to do copy/paste operation in xterm,
a) highlight the text using mouse
b) use middle button/scroll wheel of mouse to paste or
c) shift+insert to paste or when there is no middle button,or
d) simultaneously click both mouse buttons to emulate middle-button click
Thanks to Ubuntu Forums for this help.
4) All along I have been using pdftops - psselect - ps2pdf combination to select pages from a pdf document, but Today (Jan 23, 2013) I learnt about pdftk from
LinuxJournal to select pages from pdf document directly without converting to ps first. How nice!!
pdftk A=100p-inputfile.pdf cat A22-36 output outfile_p22-p36.pdf
Input file is 100p-inputfile.pdf. Though the name sounds confusing, there is no need to prefix the name with 100p (100 pages). It is just for this particular example. This command selects pages from 22 to 36 and creates an output file with just those pages.
5) April 17, 2013: Found this tip while wanting to put all docx files of submitted student homeworks in one directory to upload to google drive and open them there in order not to miss pictures or other openoffice incompatible things.
Thanks to Nixcraft I could do this and this is the first time I really found a way to use xargs command.
find . -name "*.docx" -print0 | xargs -0 -I file mv file onlydocs/
6) To sync a local directory to a remote directory:
rsync -r -a -v -e "ssh -l yourname" local/directory/path remote.machine.address:remote/directory/path
You will be prompted to enter the password to the remote machine. If all else is correct, this should sync the files. Thanks to Nixcraft for this tip.
Thursday, July 5, 2012
More python
Well, in less than one day, I came to know about two important python based software systems:
1) scikit-learn : thanks to my surfing, profile creation and eventual posting of a question in Stack-Overflow, I got to know this nice tool when I was trying hard for better python based ML tools.
2) Cython came as a surprise when I was reading about the efficiency of graph algorithms in Networkx. So, there is a way out if my experiments are going to take lot of time, which I am expecting to happen. Hopefully I should have time to fiddle with Cython.
1) scikit-learn : thanks to my surfing, profile creation and eventual posting of a question in Stack-Overflow, I got to know this nice tool when I was trying hard for better python based ML tools.
2) Cython came as a surprise when I was reading about the efficiency of graph algorithms in Networkx. So, there is a way out if my experiments are going to take lot of time, which I am expecting to happen. Hopefully I should have time to fiddle with Cython.
Monday, June 18, 2012
Life with Python
Well, though I am introduced to Python more than 6 months back, my day-to-day coding life is taking a turn towards Python direction and hence this post.
I am really tired of referring to Google every time I need to do something in Python. Phew!! how I wish I was introduced to Python in formal way! I am sure this is definitely not a good way of learning a language. Okay, acknowledgements first. I found fix to my problem here: Wrongsideofmemphis
And my issue was:
I was parsing DBLP XML file and could do it successfully with Expat library, but the problem was the parser always wrote the output to stdout. I had to do further processing of the output and thus require it to be redirected to a variable.
I am really tired of referring to Google every time I need to do something in Python. Phew!! how I wish I was introduced to Python in formal way! I am sure this is definitely not a good way of learning a language. Okay, acknowledgements first. I found fix to my problem here: Wrongsideofmemphis
And my issue was:
I was parsing DBLP XML file and could do it successfully with Expat library, but the problem was the parser always wrote the output to stdout. I had to do further processing of the output and thus require it to be redirected to a variable.
Saturday, February 18, 2012
Bayesian Multi-Task Reinforcement Learning
Authors: Alessandro Lazaric, Mohammad Ghavamzadeh
Conference: ICML, 2010
Conference: ICML, 2010
Summary:
The paper talks about
multi-task Reinforcement Learning(RL) in an environment where the
number of samples for a given task is limited in number because of
the policy involved. This work assumes that the tasks share a similar
structure and hence the corresponding value functions (vfs) are
sampled from a common prior. Because of this assumption, the authors
are able to do joint learning of vfs, in both cases of vfs from same
task class or not. The paper stand out from others in its usage of
Hierarchical Bayesian approach to model the distribution over vfs in
parametric and non-parametric setting.
Strengths:
- Generative models and inference algorithms for both cases of learning (symmetric and asymmetric) considered.
- Modeling of value function similarity by HBM.
- Different modes of learning: symmetric parametric and asymmetric non-parametric learning
- Almost all the key machine learning areas like regression, Sampling, Bayesian modeling, Expectation Maximization, Dirichlet Process etc are touched upon here making it a paper with sound theoretical arguments.
- Transfer of information from the joint distribution of vfs to learn the value function for new task.
Weaknesses:
- Authors have compared three paradigms of STL, MCMTL and SCMTL but failed to compare, on the benchmark problems, how the other related techniques perform (given that they have quoted considerable number of related works) or even further, since the authors have significantly adapted ideas from literature, they could have given a comparison of BMTL with already published results.
- The sampling techniques are computationally expensive and they are employed for asymmetric settings. Discussion of time complexity would have helped.
- The paper appeared to be an amalgamation of already established techniques, combining them in some new combination and hence it had frequent referrals to old papers for all important parameters and results which made its reading hard. In that sense, the paper is not self contained.
- No clear experimental setup to corroborate the ability to handle undefined number of classes.
- It is surprising to see that the performance dips in all cases when the number of samples increase. While it is good to see that for limited samples and increase in number of tasks, the methods do well, the proposed method should be improved to take into account large number of samples, if available.
Next steps/Discussion:
- Referring to figure 5c, it would be good to have discussion about why MCMTL fails when the number of tasks is limited in number.
- It is clear that there is some kind of transfer learning happening while learning the value function of a newly observed task. It would be interesting to analyze under what paradigm of transfer learning this paper falls into.
- It would be useful to know types of features usually considered for representing vfs in RL, esp for benchmark problems like inverted pendulum.
- Since RL is predominantly used in Robotics, it would be good to know a real world example where the vfs are from same prior.
- How is simple Gaussian processes different from GPTD?
Labels:
bayes,
critique,
ICML,
learning,
multi-task,
musings,
reinforcement
Subscribe to:
Posts (Atom)