Centralised logging is great until the incident is about the logging pipeline, or the question is too specific for the dashboard someone built last year. At that point an on-call engineer with SSH access and a few text-processing commands can answer "which clients are getting 502s, since when, on which endpoint?" faster than anyone can write a query in a web UI.

This post is the toolkit I keep in my head: a small set of commands and patterns that cover most log questions. Every example runs against the same two sample files, so you can see exactly what each step does.

The sample logs

An nginx access log in the default combined format plus request time:

203.0.113.7 - - [06/Jun/2025:09:14:02 +0000] "GET /api/orders?page=2 HTTP/1.1" 200 1532 "-" "okhttp/4.12" 0.084
198.51.100.23 - - [06/Jun/2025:09:14:02 +0000] "POST /api/checkout HTTP/1.1" 502 166 "-" "Mozilla/5.0" 30.001
203.0.113.7 - - [06/Jun/2025:09:14:03 +0000] "GET /api/orders?page=3 HTTP/1.1" 200 1488 "-" "okhttp/4.12" 0.091
192.0.2.55 - - [06/Jun/2025:09:14:05 +0000] "POST /api/checkout HTTP/1.1" 502 166 "-" "Mozilla/5.0" 30.002

And an application log with one JSON object per line:

{"ts":"2025-06-06T09:14:02.118Z","level":"error","msg":"payment timeout","order_id":"o_9921","provider":"acme-pay","duration_ms":30000}
{"ts":"2025-06-06T09:14:02.410Z","level":"info","msg":"order listed","user_id":"u_17","duration_ms":61}

grep: find the lines

grep ' 502 ' access.log | head                # lines containing a 502 status
grep -c ' 502 ' access.log                    # just the count
grep -E '" (5[0-9]{2}) ' access.log | wc -l   # all 5xx
grep -v 'healthcheck' access.log              # everything except health checks
grep -i 'timeout' app.log                     # case-insensitive
grep -B2 -A5 'OutOfMemory' app.log            # context before and after

Two habits save time. First, anchor patterns tightly: ' 502 ' with spaces avoids matching a byte size of 1502. Second, use zgrep for rotated, compressed files, so you do not unpack gigabytes:

zgrep -h ' 502 ' /var/log/nginx/access.log.*.gz | wc -l

When the log is large, grep -F for fixed strings is noticeably faster than a regex, and LC_ALL=C speeds up most of these tools on plain ASCII logs.

awk: pick and compute fields

awk splits each line on whitespace into $1, $2 and so on. In the combined format, the client IP is $1, the status $9, the bytes $10 and our request time is the last field, $NF.

awk '{print $9}' access.log | sort | uniq -c | sort -rn

   2 502
   2 200

Requests slower than one second, with time, path and duration:

awk '$NF > 1 {print $4, $7, $NF}' access.log

[06/Jun/2025:09:14:02 /api/checkout 30.001
[06/Jun/2025:09:14:05 /api/checkout 30.002

Average and maximum request time per status:

awk '{sum[$9]+=$NF; n[$9]++; if ($NF>max[$9]) max[$9]=$NF}
     END {for (s in sum) printf "%s avg=%.3f max=%.3f n=%d\n", s, sum[s]/n[s], max[s], n[s]}' access.log

200 avg=0.088 max=0.091 n=2
502 avg=30.002 max=30.002 n=2

The 30-second durations on every 502 point at an upstream timeout, which is a far more useful lead than "some 502s".

sort and uniq: count and rank

The pattern sort | uniq -c | sort -rn is the single most useful pipeline in log analysis. It turns any column into a ranked frequency table.

Top client IPs by request count:

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head

Top paths without query strings:

awk '{split($7, p, "?"); print p[1]}' access.log | sort | uniq -c | sort -rn | head

Errors per minute, to see when a problem started:

grep ' 502 ' access.log | awk '{print substr($4, 2, 17)}' | uniq -c

   2 06/Jun/2025:09:14

uniq only collapses adjacent duplicates, so it needs sorted input, except when the input is already in time order and you want counts per time bucket, as in the last example.

cut and tr: quick slicing

For fixed delimiters, cut is simpler than awk:

cut -d' ' -f1 access.log | sort -u | wc -l     # unique client IPs
cut -d'"' -f6 access.log | sort | uniq -c       # user agents (6th field when splitting on quotes)
tr -s ' ' < messy.txt                           # squeeze repeated spaces before cutting

jq: structured JSON logs

jq reads JSON and lets you filter and reshape it. With one object per line, every example below processes the file as a stream:

jq -c 'select(.level == "error")' app.log
jq -r 'select(.level == "error") | [.ts, .msg, .order_id] | @tsv' app.log
jq -r 'select(.duration_ms > 1000) | .provider' app.log | sort | uniq -c | sort -rn

      1 acme-pay

Group and count inside jq with slurp mode (-s), which reads the whole file into memory; fine for thousands of lines, not for gigabytes:

jq -s 'map(select(.level=="error")) | group_by(.msg) | map({msg: .[0].msg, n: length})' app.log

If some lines are not valid JSON, for example a stack trace printed raw, filter first:

grep '^{' app.log | jq -c 'select(.level=="error")'

Following a live log

tail -F /var/log/nginx/access.log | grep --line-buffered ' 502 '

-F keeps following through log rotation. --line-buffered makes grep print each match immediately instead of waiting to fill a buffer, which matters when you pipe into another command.

Putting it together: a real question

Question: during the checkout incident, which clients were affected, how many times, and when did it start and stop?

zgrep -h 'POST /api/checkout' /var/log/nginx/access.log* \
  | awk '$9 >= 500 {print $1, substr($4, 2, 17)}' \
  | sort -k2 \
  | awk '{c[$1]++; if (!first) first=$2; last=$2} END {print "first", first, "last", last; for (ip in c) print c[ip], ip}' \
  | sort -rn | head

One pipeline gives the time window and the most affected clients, which is what the incident channel asked for.

Common mistakes

Mistake Why it hurts Do this instead
cat file | grep on huge files Extra process, no benefit grep pattern file
Unanchored patterns like 502 Matches sizes and IDs Include delimiters: ' 502 '
uniq -c on unsorted input Counts are split sort | uniq -c
Assuming field numbers in custom formats Wrong column silently Check log_format first, count fields with awk '{print NF}'
jq -s on gigabyte logs Loads everything in memory Stream with select, pre-filter with grep
Running heavy pipelines on a struggling server Adds load during an incident Copy logs elsewhere, or use nice

A cheat sheet to keep

count lines            grep -c PATTERN file
ranked frequency       ... | sort | uniq -c | sort -rn | head
field N                awk '{print $N}'      last field: $NF
numeric filter         awk '$NF > 1'
per-minute buckets     awk '{print substr($4,2,17)}' | uniq -c
compressed logs        zgrep -h PATTERN file*.gz
JSON filter            jq -c 'select(.level=="error")'
JSON to TSV            jq -r '[.a,.b] | @tsv'
live follow            tail -F file | grep --line-buffered PATTERN

Key takeaways

  • grep finds lines, awk extracts and computes fields, sort | uniq -c | sort -rn ranks anything.
  • Anchor patterns and know your log format's field positions before trusting numbers.
  • jq with select handles structured logs as a stream.
  • Use zgrep and tail -F to cover rotated and live logs.
  • Build the answer in small steps, checking each stage's output before adding the next.

References