Awk and Sed

1) To search and replace in vi, you need not enter inside the file at all:
sed does the magic for you from command prompt itself. Tip here.

2) To alter the order of the columns in a file and write the output to another file:
awk '{print $3" "$1" "$2}' filename > newfilename

The above example takes a 3-columned file and changes the order as 1st col as 2nd, 2nd as 3rd and 3rd as 1st and writes the output to newfilename. There is no space between the double quotes and column indices. $n stands for specific column in the file in awk terminology. I am not sure if giving same source filename as the destination is going to work.


3) To check if the first column of a line in a file is some character and if yes, print the entire line:
awk '{if ($1~/e/) print $0}' filename

Here the first column has only one character throughout. $1 indicates the first column. In this specific example, if the first column contains a single character 'e', the command prints the entire line, indicated by $0. The single character can be replaced by a word too and there is no need for quotes around the word or character to be searched. There is no need for space around '~' character.

The default delimiter is space character. If you want to give a different delimiter, add -F\<delimter>. If the field you are comparing is a number, can give numeric relational operators in if condition like ==, !=, > etc.

4) Similar to case3 above, but to give more than one condition in the if  statement:
awk '{if (($1 ~ /e/) || ($1~/b/)) print $0}' filename

There are two important differences between case3 and case4. There are now enclosing '/' present around characters to be searched for and there is a logical OR sign '||'.

5) To search and replace character(s) from command prompt on a bunch
of files :

for((i=1;i<=30;i++)); do
sed -e 's/t:/ /g' filename$i > newfilename$i ; done

For 30 files that all have common prefix, search for character 't:' and replace them with a single space and write the output to a newfile with the same number

6)  Aug 13, 2013: To add a column of numbers generated as a result of say some intermediate linux util commands,

grep 'nonworkingset' tobedel|cut -f2 -d:|awk '{sum+=$1} END {print sum}'


The first two commands take care of getting just the list of numbers. Note that if capitalize sum then you should do it uniformly across the entire command.

Linux Util 1

1) To paste one of the columns of a multi-column file as one other column in the same file:
cut -f3 -d' ' fname1|paste -d' ' - fname1 > fname2

This command takes the 3rd column of file fname1 and pastes it as first column and writes the content to file fname2. Rename the file fname2 to fname1 to avoid keeping duplicate copies. The hyphen in the paste command can be switched to last to make the 3rd column as the last column instead of first. Here the column separator is the space character given after the '-d' switch.


2) I forget to leave space around binary operator used in expr command in shell scripting.

3) I  keep forgetting about the rules of double quotes and single quotes usage in shell scripting. If you give a single quoted word inside double quoted sentence, the single quote is not interpreted, but a $variable is substituted with its value.

4) To find all C/C++ program files and related header files:

find . -regex '.*\.\(c\|h\|cxx\|cpp\)' -print

This command fetches files with extension .c, .h, .cxx, .cpp and prints their path from the current directory. Please note the extra characters ".*." in the front. The second dot is escaped and so is the first open parenthesis and logical OR symbol '|'.

5) A modification to the above command is by replacing the last 'print' option to 'print0'. That is append number zero to print word. This will display all the file names in one line as against the previous output that prints one file name in one line.

6) If you want the contents of all these files in one go, give the following:

find . -regex '.*\.\(c\|h\|cxx\|cpp\)' -print0|xargs -o cat

7) To remove all spaces in the filenames of a directory, get inside the directory and type in command line:

rename 's/ //g' *

Here, the single space after first '/' is important. Replace '*' with other names if you do not want to apply this change to all files.