Linux Util 6

I inadvertently deleted the previous contents of this post. Blogger sucks :( would not allow me to revert the contents. I lost one of my precious search on the nmap command.

1) To add a column of numbers and print the sum:
 
awk '{sum += $2} END {print sum}' temp 

2) To list the contents of a local directory when inside a ftp connection:

!ls

In words, exclamatory sign followed by the actual command.

3) If you are transferring a lot of files through mput or mget and do not have the patience to hit the key 'y', then while opening the ftp connection:

ftp -i hostname

Interestingly, I would have thought -i would enable interactive mode, here is just the reverse.

4) If you have to convert from Unix time stamp to human-readable date form and vice versa, the following helps:

date -d @915149280       (to get readable date format from epoch seconds)

date +%s -d"Thu Nov 1 12:50:00 2013"         (to get the unix timestamp in seconds from normal date format)

I literally stumbled upon the second command while reading a related question and its answer from SO. This answer was both simple and complicated in its own way. I thought I am better with vim search and replace. Well, I thought wrong. So, here is the next tip. Stack Overflow rocks!!

5) If you have text file with full human-readable date string sitting between the columns, then use vim to replace them all to epoch time using the following search string.

An example line looks like this:
Jey tty8 Thu Nov 1 12:50:57 2012 - Thu Nov 1 12:51:21 2012 (00:00)

Search command (in escape mode in vim):
:%s/\v\w+\s\w+\s\d+\s\d+:\d+:\d+\s\d+/\=system('date +%s -d"'.submatch(0).'" | tr -d "\n"')/g

After replacing:
(Jey tty8 1351788657 - 1351788681 (00:00)

As you can see, after the 'system' word, sits the command I explained in point 4 above. I learnt a lot of new things from this search regular expression.
a) use of the submatch(0): this matches what you went looking for in the first half of the expression
b) tr is used for translation. In this case, it is used to delete the trailing "\n" that comes with executing the system command
c) \s is for matching a single space. What was surprising was the "+" symbol after the characters 'w' and 'd'. Without them, the command does not work.

Linux util 5

Whew!! My 5th linux util post...

1) Assuming you have SLURM basics, (well, I am just a beginner, so posting this very simple solution for the problem I faced), to force salloc to release the job allocation, find the process of that command using

   a) ps ax|grep $your_name|grep salloc

   If you have reserved resources by separate salloc command (without invoking the job using srun), then
   b) kill -9 $pid

Or, If you have both salloc and srun in a single command,
  c) kill -s HUP $pid

To see the signal identification for all the signals, use

man 7 signal

Just 'man signal' will take you to the C programming API for signaling.

2) Here is a simple way to find the number of words in each line of a file using awk:

awk '$0="line"NR": "NF' filename

Guess I got this tip again from SO, but this solution was not the accepted one.

3) If you pick only few lines output from make command to selectively fix the errors or warnings, (assuming you are in bash shell):


 make -f Makefile clean; make -f Makefile 2 > &1 |grep 'error'

4) To identify the shell you are working on currently:
 ps -p $$
From what I understand from the man pages, the -p option looks for pid list and $$ get the first process which is the shell itself. This prints the pid of the shell, terminal id etc. If you would rather want a concise output, then type:
echo $0 
This was important to me change between bash and tcsh frequently when working on HPC machines of PNNL. The default login shell was tcsh, but I had do compilation and execution on bash. Most importantly, echo $SHELL did not help.

Awk and Sed

1) To search and replace in vi, you need not enter inside the file at all:
sed does the magic for you from command prompt itself. Tip here.

2) To alter the order of the columns in a file and write the output to another file:
awk '{print $3" "$1" "$2}' filename > newfilename

The above example takes a 3-columned file and changes the order as 1st col as 2nd, 2nd as 3rd and 3rd as 1st and writes the output to newfilename. There is no space between the double quotes and column indices. $n stands for specific column in the file in awk terminology. I am not sure if giving same source filename as the destination is going to work.


3) To check if the first column of a line in a file is some character and if yes, print the entire line:
awk '{if ($1~/e/) print $0}' filename

Here the first column has only one character throughout. $1 indicates the first column. In this specific example, if the first column contains a single character 'e', the command prints the entire line, indicated by $0. The single character can be replaced by a word too and there is no need for quotes around the word or character to be searched. There is no need for space around '~' character.

The default delimiter is space character. If you want to give a different delimiter, add -F\<delimter>. If the field you are comparing is a number, can give numeric relational operators in if condition like ==, !=, > etc.

4) Similar to case3 above, but to give more than one condition in the if  statement:
awk '{if (($1 ~ /e/) || ($1~/b/)) print $0}' filename

There are two important differences between case3 and case4. There are now enclosing '/' present around characters to be searched for and there is a logical OR sign '||'.

5) To search and replace character(s) from command prompt on a bunch
of files :

for((i=1;i<=30;i++)); do
sed -e 's/t:/ /g' filename$i > newfilename$i ; done

For 30 files that all have common prefix, search for character 't:' and replace them with a single space and write the output to a newfile with the same number

6)  Aug 13, 2013: To add a column of numbers generated as a result of say some intermediate linux util commands,

grep 'nonworkingset' tobedel|cut -f2 -d:|awk '{sum+=$1} END {print sum}'


The first two commands take care of getting just the list of numbers. Note that if capitalize sum then you should do it uniformly across the entire command.