"The server is slow" is the least specific bug report in operations, and the first few minutes after it decide whether you find the cause or spend an hour staring at the wrong graph. There are dozens of Linux performance tools. You only need a handful, run in a sensible order, to tell the four main bottlenecks apart: CPU, memory, disk and network.

This is the routine I use on call. It takes about ten minutes, needs only packages present on most distributions (procps, sysstat, iproute2), and every step narrows the next one.

Minute 0-1: what does "slow" mean, and since when?

Before touching the server, get two facts: which operation is slow (every request, one endpoint, a batch job) and when it started. A deploy, a cron job or a traffic change at that time is often the answer. Then log in and get context:

uptime

 03:12:40 up 41 days,  2:01,  1 user,  load average: 18.42, 15.10, 6.33

Load average counts tasks that are runnable or in uninterruptible sleep (usually waiting for disk). On Linux a high load can mean CPU saturation or I/O waiting; it does not say which. Compare with the number of CPUs (nproc) and look at the trend: 18 now, 6 fifteen minutes ago means something recent.

dmesg -T | tail -20

Kernel messages reveal OOM kills, disk errors, network link flaps and hung tasks in seconds. Skipping this step is the most common reason people miss the real cause.

Minute 1-2: pressure stall information

On kernels 4.20 and newer, PSI tells you directly how much time tasks spend stalled waiting for each resource:

cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io

some avg10=12.40 avg60=10.98 avg300=4.20 total=...
some avg10=0.00 avg60=0.00 avg300=0.00 total=...
full avg10=0.00 avg60=0.00 avg300=0.00 total=...
some avg10=61.20 avg60=55.10 avg300=20.71 total=...
full avg10=48.90 avg60=44.30 avg300=16.02 total=...
  • some is the share of time at least one task was stalled on that resource.
  • full is the share of time all non-idle tasks were stalled at once, meaning no useful work.

Here I/O pressure is very high (full 48.9% over the last 10 seconds) while memory is zero. That points straight at storage. PSI is the fastest single signal for which resource to investigate.

Minute 2-4: vmstat, the system overview

vmstat -w 1 5

procs -----------memory---------- ---swap-- -----io---- -system-- --------cpu--------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2 14      0 812340  10204 9123400   0    0 18204 92110 4120 6230  6  4 22 68  0
 3 15      0 809112  10204 9124520   0    0 17988 95004 4210 6410  5  4 20 71  0

The columns that matter:

Column Meaning Worry when
r Runnable tasks Consistently above CPU count: CPU saturation
b Tasks blocked in uninterruptible sleep Non-zero and climbing: usually I/O
si, so Swap in/out per second Any sustained non-zero: memory pressure
bi, bo Blocks read/written per second High with high wa: disk bound
us, sy User and kernel CPU us+sy near 100: CPU bound
wa CPU idle waiting for I/O High: storage is the bottleneck
st Time stolen by the hypervisor Non-zero on VMs: noisy neighbour or exhausted burst credits

Ignore the first line; it shows averages since boot. In this output, b is 14, wa near 70% and bo high: processes are waiting for writes.

Quick reading guide:

  • High r, high us, low wa: CPU. Go to pidstat -u and top.
  • Non-zero si/so, low free and small cache: memory. Go to free -m and per-process RSS.
  • High b, high wa: disk. Go to iostat.
  • Everything low but requests are slow: look outside the box, at network or dependencies.

Minute 4-6: iostat, which device and how bad

iostat -xz 1 3

Device   r/s     w/s    rkB/s    wkB/s  r_await  w_await  aqu-sz  %util
nvme1n1  12.0  1840.0   480.0  94210.0     1.20    38.90   71.40  100.00
nvme0n1   3.0    14.0    40.0    210.0     0.40     0.80    0.02    1.20
  • r_await, w_await: average time in milliseconds a request spent queued plus being served. For NVMe, single-digit values are normal; 39 ms of write latency is not.
  • aqu-sz: average queue length. Large values mean requests are piling up.
  • %util: share of time the device was busy. For modern SSDs that handle many requests in parallel, 100% does not always mean saturated; trust await and aqu-sz more.

The data volume nvme1n1 is overloaded with writes. On cloud volumes, also check whether you have hit the provisioned IOPS or throughput limit, which looks exactly like this.

Minute 6-8: pidstat, which process

Now find who is doing it:

pidstat -d 1 5      # disk I/O per process
pidstat -u 1 5      # CPU per process
pidstat -r 1 5      # memory faults and RSS per process

03:14:02 PM   UID   PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
03:14:03 PM   999  4122      0.00  88104.00      0.00     212  postgres
03:14:03 PM     0  9310      0.00   5120.00      0.00      18  rsync

PostgreSQL is writing 88 MB/s. That turns "the server is slow" into a specific question for the database: checkpoint, vacuum, a bulk import or a query spilling to disk? pidstat -d is often the step that ends the investigation.

For threads within a process, add -t. For a single process over time, -p PID.

Minute 8-9: memory, if vmstat pointed there

free -m

               total        used        free      shared  buff/cache   available
Mem:           31890       29812         410          88        1668        1402
Swap:           4095        3890         205

Look at available, not free. Linux uses spare memory for cache, so low free is normal. Low available plus swap activity in vmstat means real pressure. Then check which processes hold memory and whether the OOM killer has fired:

ps -eo pid,rss,comm --sort=-rss | head
journalctl -k | grep -i 'out of memory'

Minute 9-10: network and dependencies

If the box looks healthy, the slowness is often in what it talks to:

ss -s                       # socket summary: established, time-wait, orphaned
ss -tin state established '( dport = :5432 )' | head   # rtt and retransmits to the DB
sar -n DEV 1 3              # interface throughput
sar -n EDEV 1 3             # interface errors and drops

Rising retransmits, a pile of connections in SYN-SENT or CLOSE-WAIT, or a saturated interface each tell a different story: network loss, an unreachable dependency, an application not closing sockets, or bandwidth exhaustion.

Common mistakes

Mistake Why it hurts Do this instead
Reading load average as CPU usage On Linux it includes I/O wait Check vmstat r vs b and PSI
Trusting the first line of vmstat or iostat It is the since-boot average Read from the second sample
Panicking about low "free" memory Cache is reclaimable Use the available column
Treating 100% util on SSD as saturation SSDs work in parallel Look at await and queue size
Skipping dmesg Misses OOM kills, disk errors Always read recent kernel messages
Running heavy tools on a dying box Makes it worse Start with cheap commands, sample briefly

Make it repeatable

Turn the routine into a script that collects everything in one go, so the next person on call gets the same evidence:

#!/usr/bin/env bash
set -u
out="/tmp/triage-$(hostname)-$(date +%Y%m%dT%H%M%S).txt"
{
  uptime; nproc; dmesg -T | tail -30
  cat /proc/pressure/{cpu,memory,io}
  vmstat -w 1 5; iostat -xz 1 3; free -m
  pidstat -u -d -r 1 3
  ss -s; sar -n DEV 1 3
} > "$out" 2>&1
echo "saved $out"

Keep sysstat collection enabled (sadc via its timer) so you can look at the past with sar when the problem has already stopped.

Key takeaways

  • Ask what is slow and since when, then read uptime and dmesg first.
  • PSI shows in seconds which resource tasks are stalled on.
  • vmstat tells CPU, memory and I/O apart; ignore its first line.
  • iostat -x latency and queue size matter more than %util on SSDs.
  • pidstat names the process, which turns a vague complaint into a specific question.

References