awk Examples

gawk - pattern scanning and processing language

gawk [ POSIX or GNU style options ] -f program-file [ -- ] file ...
gawk [ POSIX or GNU style options ] [ -- ] program-text file ...

-f program-file
-F field separator (FS)
-v var=val Assign the value val to the variable var, before execution of the program begins. Such variable values are available to the BEGIN rule of an AWK program.

Patterns
   AWK patterns may be one of the following:

      BEGIN
      END
      BEGINFILE
      ENDFILE
      /regular expression/
      relational expression
      pattern && pattern
      pattern || pattern
      pattern ? pattern : pattern
      (pattern)
      ! pattern
      pattern1, pattern2

• projection on column one and four
  awk -F, '{print $1, $4}' ../data/city.sample.csv
  [Output]

• returns all bavarian cities with less than a million inhabitants
  awk -F, '$3=="Bayern" && $4 < 1000000 { print $1", "$4 }' ../data/city.sample.csv
  [Output]

• calculates the average population of all cities (read awk script from file -f option)
  awk -F, -f ./average.awk ../data/city.sample.csv
  [Output]

• wrong splitting of input record (comma inside "...")
  awk -F, '{print $1" : "$3}' fpat-test.txt
  [Output]

• correct field splitting, based on FPAT pattern set (specify field pattern instead of separator pattern)
  awk 'BEGIN{FPAT = "([^,]*)|(\"[^\"]*\")"} { print $1" : "$3}' fpat-test.txt
  [Output]

• write the cities for every country (specified by column 2) in a separate file
  rm -f tmp/cities-*.csv && awk -F, '{print $0 >> "tmp/cities-"$2".txt"}' ../data/city.sample.csv
  [Output], generated files: [tmp/cities-D.txt, tmp/cities-E.txt, tmp/cities-F.txt, tmp/cities-I.txt]

• example with parameter-passing and input from STDIN
  echo -e "line 1\nline 2\nline 3" | awk -vtable="A header line" 'BEGIN { print table } { print $0 }'
  [Output]

• definition of a shell function calc, based on awk and use of it
  calc() { awk "BEGIN{ print $* }" ;} && calc 1+5*6
  [Output]

• execute different action, depending on the value ('next' ends the current iteration)
  seq 1 31 | awk '$1 % 2==0 {printf "%03d is even\n", $1; next} {print $1" is sooo odd"}'
  [Output]

• extract a random 0.01% sample from dataset ../data/BCHI-dataset_2019-03-04.txt
  awk 'rand() < 0.0001 {print $0}' ../data/BCHI-dataset_2019-03-04.txt
  [Output]

• extract a sample of cities with more then 100000 inhabitants from Germany, France, Italy, Spain and Suisse
  awk -F, '$2~/^(D|F|I|CH|E)$/ && $4!="NULL" && $4 > 100000 && rand() < 0.5 {print $0}' ../data/city.sample.csv
  [Output]