Awk and Sed

1) To search and replace in vi, you need not enter inside the file at all:
sed does the magic for you from command prompt itself. Tip here.

2) To alter the order of the columns in a file and write the output to another file:
awk '{print $3" "$1" "$2}' filename > newfilename

The above example takes a 3-columned file and changes the order as 1st col as 2nd, 2nd as 3rd and 3rd as 1st and writes the output to newfilename. There is no space between the double quotes and column indices. $n stands for specific column in the file in awk terminology. I am not sure if giving same source filename as the destination is going to work.


3) To check if the first column of a line in a file is some character and if yes, print the entire line:
awk '{if ($1~/e/) print $0}' filename

Here the first column has only one character throughout. $1 indicates the first column. In this specific example, if the first column contains a single character 'e', the command prints the entire line, indicated by $0. The single character can be replaced by a word too and there is no need for quotes around the word or character to be searched. There is no need for space around '~' character.

The default delimiter is space character. If you want to give a different delimiter, add -F\<delimter>. If the field you are comparing is a number, can give numeric relational operators in if condition like ==, !=, > etc.

4) Similar to case3 above, but to give more than one condition in the if  statement:
awk '{if (($1 ~ /e/) || ($1~/b/)) print $0}' filename

There are two important differences between case3 and case4. There are now enclosing '/' present around characters to be searched for and there is a logical OR sign '||'.

5) To search and replace character(s) from command prompt on a bunch
of files :

for((i=1;i<=30;i++)); do
sed -e 's/t:/ /g' filename$i > newfilename$i ; done

For 30 files that all have common prefix, search for character 't:' and replace them with a single space and write the output to a newfile with the same number

6)  Aug 13, 2013: To add a column of numbers generated as a result of say some intermediate linux util commands,

grep 'nonworkingset' tobedel|cut -f2 -d:|awk '{sum+=$1} END {print sum}'


The first two commands take care of getting just the list of numbers. Note that if capitalize sum then you should do it uniformly across the entire command.

vi tips

1) To convert a dos text file to unix file, in the escape mode of vi editor, type:

:set ff=unix

This removes the additional carriage return "\r" at the end of each line

2) To search and replace, in the escape mode:
:.,$s/pat1/pat2/g

This replaces pat1 by pat2 in all occurrences starting from the current position of the character to the end of the file. The current position indicated by . after : and end of the file is given by $ sign after the comma.

3) In command mode, to move to the beginning of the next line, type '+' and '-' to move to the starting of the previous line

4) To remove space from the beginning of all lines in vi editor:

:%s/^b\+//g

5) To replace a bunch of characters to newline in vi editor, use \r for searching.
For example, (1,2,3), ('a','b','c'), ('.....) : this list of tuples all printed in one line can be split as
(1,2,3)
('a','b','c')
....
by typing in escape mode in vi editor:

:%s/), /\r/g

Note the space following ), characters. Note also that there is no need to escape \ in \r.

5) So, during the initial days of my cooking, I used to just  copy and paste the recipes from web to my editor. This editor was Notepad during my W7 days and when I installed Mint 16, I naturally switched to vi. Because of this transition, I started seeing a lot of Windows specific metacharacters in the recipe file. They look like this: <96> or <97>. You cannot search for this combination of characters as they are not separate. For this, I found the solution from SuperUser by this user. 

:%s/[\x96]//g

This syntax deletes all the occurrences. You can add additional characters as follows:

:%s/[\x96\x97]//g

It is important to have \x in the syntax to indicate you are searching for special characters in their hex code. If you know the exact equivalent character you may add it in the 'replace' part of the command.