In tonight's episode of "Fun with pipelines", we've got this monstrosity:
awk '{ print $3 }' logs/* |
sort -n |
uniq |
tail +6 |
awk '{print "-x " $1 }' |
xargs dig +noall +answer |
rev |
awk -F. '{print $2 "." $3}' |
rev |
sort |
uniq -c |
sort -nr |
head -n 10
which produces an output something like:
349 amazonbot.amazon
186 ahrefs.net
67 yandex.com
47 msn.com
39 semrush.com
16 petalsearch.com
14 babbar.eu
10 googleusercontent.com
10 google.com
telling us that amazon used nearly 350 different source IPs to make requests to this server.
Explanation
It's bash, so the | at the ends of the lines means "pass the output of this command to the next command".
awk '{ print $3 }' logs/*
Awk is a small text processing tool. By default, it splits its input into lines (using \n) and then into fields (using spaces and tabs). In this case, print $3 means "print the 3rd (counting from 1) field". We've told awk to read all the files in the logs folder, which have an IP address in the 3rd field.
sort -u
sort takes it's input, splits into lines, sorts it, and prints the output. The -u option tells sort to only output one copy of identical lines (which is what we really want, the sorting is a side effect).
awk '{print "-x " $1 }'
Awk again, this time printing each input line with "-x " in front, which helps us build the next command.
xargs dig +noall +answer
xargs is a neat toy. It takes the input fed to it, and the command you give it, and then it runs the command, passing the command all the input as options.
It's very useful for, well, this kind of situation, where we've got a list of things that we want to run a command with.
The command we've asked it to run is dig, a "DNS lookup utility". Given a hostname, it will find the IP, or (as in our case), given an IP address, it will find the hostname. We have to give it the -x option per address to tell it we're looking for the name. the +noall +answer options limit it's output to the part we care about (the answer!).
rev
rev outputs each line of input reversed.
(Sorry, I ran out of energy at this point. Hopefully I'll come back and finish at some point)