Useful one liner awk, perl, sed

Very useful handy 1 liners needed to be successful, good Administrator. Its very handy to troubleshoot issue Smartly.

Sed (stream editor): Sed id a stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline).

AWK (Aho, Weinberger, and Kernighan):  is a programming language designed for text processing and typically used as a data extraction and reporting tool. It is a standard feature of most Unix-like operating system

Perl (Practical Extraction and Report Language):

Perl is a powerful text processing facility provider that makes report processing easier. 


Below few one liner we can use in day to day Linux shell programming and some string management:


1) find and Replace in file
perl -pi -e 's/first/second/g' file1 file2 file3

2) find and replace only on line matching word "single"
perl -pi -e 's/first/second/g if /single/' file

perl -ne 'print if $a{$_}++' file

3) Print the N records after some pattern:

awk 'c&&c--;/pattern/{c=N}' file


4) Kill all process matching pattern

kill -9 $(ps -ef|grep search_string|grep -v grep|awk -F " " '{ print $2 }')

kill -9 `ps -ef |awk ' /search_string/ {print $2} '`

5) delete ALL blank lines from a file (same as "grep '.' ")
 awk NF
 awk '/./'

6) remove duplicate, nonconsecutive lines
 awk '!a[$0]++'                     # most concise script
 awk '!($0 in a){a[$0];print}'      # fast

7) print the line immediately after a regexp, but not the line
 # containing the regexp
 sed -n '/regexp/{n;p;}'

7) print 1 line of context before and after regexp, with line number
 # indicating where the regexp occurred (similar to "grep -A1 -B1")
 sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h


8) print section of file based on line numbers 
 sed -n '18,121p'               # way 1
 sed '18,121!d'                 # way 2

9) awk condition Check
YESTERDAY=$(date --date "1 days ago" +%d%m%y)
DATAFILE=visitor_log_$YESTERDAY.txt
#!/usr/bin/expect --
#cat $DATAFILE > VISITOR_LOG_BIG
cat DILIP_log.bad >> ALL_DILIP_log.bad

awk -F '|' '{if((length($1)==12)&&($1>910000000000)&&($1<=919999999999)) print $1,"|",$2,"|",$9}' $DATAFILE | sort | uniq >  VISITOR_LOG_BIG