Uplink saturated at 3 a.m.: finding the Linux process eating bandwidth with iftop, nethogs and ss

For two weeks our latency dashboard showed the same shape: every night between 03:00 and roughly 04:10 UTC, p95 latency of the public API tripled, then dropped back. The pattern was too regular to be traffic, and our users in Asia, for whom 03:00 UTC is mid-morning, were the ones complaining.

The API ran on four servers behind a load balancer. Three looked normal. The fourth, api-04, showed transmit throughput on its main interface pinned at the link limit during that hour.

Step 1: confirm at the interface level

sar from the sysstat package keeps historical interface counters, which is exactly what you need for something that happens while you are asleep:

sar -n DEV -f /var/log/sysstat/sa14 -s 02:50:00 -e 04:20:00 | grep -E 'IFACE|ens5'

02:50:01 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   %ifutil
03:00:01 AM      ens5   4210.33  81450.12    310.25 118904.77     97.41
03:10:01 AM      ens5   4122.80  81732.55    301.92 119310.41     97.74
04:00:01 AM      ens5   4301.11  80981.40    315.08 118223.90     96.85
04:10:01 AM      ens5   3984.20   5120.66    296.40   6012.33      4.93

About 119 MB/s out, on a 1 Gbit/s interface. Something was sending, not receiving.

Step 2: which remote host?

The next night, at 03:05, we watched live with iftop, which shows bandwidth per pair of hosts:

sudo iftop -i ens5 -n -N -P -B

                 12.5MB        25.0MB        37.5MB        50.0MB    62.5MB
10.0.4.21:41872   => 198.51.100.40:443        112MB  110MB  109MB
                  <=                           310KB  302KB  298KB
10.0.4.21:443     => 203.0.113.88:51022        210KB  205KB  198KB
...

Almost all traffic went to one external address on port 443. That address belonged to an object storage provider's endpoint.

Step 3: which process?

iftop knows hosts, not processes. nethogs groups traffic by process:

sudo nethogs ens5

  PID USER     PROGRAM                          DEV        SENT      RECEIVED
 2841 backup   /usr/bin/restic                  ens5    114210.3      284.1 KB/sec
 1107 www-data php-fpm: pool www                ens5       201.2      188.6 KB/sec

restic, our backup tool. To be certain and see the connection details, ss lists sockets with the owning process and TCP internals:

sudo ss -tinp dst 198.51.100.40

ESTAB 0 3145728 10.0.4.21:41872 198.51.100.40:443 users:(("restic",pid=2841,fd=9))
	 cubic wscale:7,7 rto:204 rtt:2.81/0.4 mss:8949 cwnd:412 bytes_sent:401227551744 ...

A large send queue and a big congestion window: a bulk upload running as fast as TCP would allow.

What we ruled out

• An attack or data exfiltration. The destination was our own backup bucket endpoint, the process was the expected backup binary running as the backup user, and its command line matched the configured job.
• Load balancer imbalance. Request counts per server were equal; api-04 was slower because its uplink was full.
• Provider network issues. The other three servers had normal latency to the same clients.

Root cause

A new backup job, added a month earlier, uploaded the full media directory from api-04 to object storage every night. It had no bandwidth limit, ran on the same interface as production traffic, and was scheduled at 03:00 UTC, which someone had picked as "the quiet hour" based on European traffic. TCP shared the link fairly between flows, but one bulk flow against hundreds of small API responses meant the API responses queued behind it.

Why one upload hurts many small requests

TCP congestion control tries to share a bottleneck fairly per flow. A single long-lived upload keeps the link and the queue in front of it full. Small API responses must wait in that queue, so each one gains tens of milliseconds of queueing delay, and anything needing several round trips (TLS handshakes, slow-start) is hit several times. Throughput looks fine; latency does not.

What each tool sees

Environment

Immediate: limit the backup's bandwidth

restic supports upload limits in KiB/s directly:

restic backup /srv/media --limit-upload 30720   # ~30 MiB/s, about a quarter of the link

We also lowered its CPU and I/O priority in the systemd unit, so it yields to the application:

[Service]
User=backup
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7
ExecStart=/usr/bin/restic backup /srv/media --limit-upload 30720

Defence in depth: shape traffic at the host

Tool-level limits depend on every job being configured correctly. As a safety net, we added traffic shaping with tc, giving the backup user's traffic a lower class. Packets are marked by owner with nftables, then classified by the mark:

# mark packets from the backup user
nft add table inet mangle
nft add chain inet mangle output '{ type route hook output priority -150; }'
nft add rule inet mangle output meta skuid backup meta mark set 0x10

# HTB: default class for everything, limited class for marked backup traffic
tc qdisc add dev ens5 root handle 1: htb default 10
tc class add dev ens5 parent 1: classid 1:1 htb rate 950mbit
tc class add dev ens5 parent 1:1 classid 1:10 htb rate 700mbit ceil 950mbit prio 0
tc class add dev ens5 parent 1:1 classid 1:20 htb rate 100mbit ceil 300mbit prio 1
tc qdisc add dev ens5 parent 1:10 fq_codel
tc qdisc add dev ens5 parent 1:20 fq_codel
tc filter add dev ens5 parent 1: protocol all handle 0x10 fw flowid 1:20

fq_codel on each class also keeps queues short, which helps latency even under load.

Move the schedule and the data path

• The job now runs at 21:00 UTC, the actual lowest point of global traffic, found from the load balancer request metrics rather than guessed.
• Media files are now uploaded to object storage by the application when they are created, so the nightly job only handles a small delta.

Alternatives and trade-offs

• A separate network interface for backups. Clean separation, but needs network design and is not available on every instance type.
• Back up from a replica or snapshot instead of the serving host. The best long-term answer for databases; for media files, the application-level upload removed the need.

Confirming in production

The following nights, sar showed transmit throughput on ens5 peaking around 30 MB/s during the backup, and API p95 latency stayed within its normal band. The backup took 40 minutes longer, which was acceptable.

tc -s class show dev ens5
# class htb 1:20 ... Sent 108742215123 bytes ... rate 245Mbit

Regression guard

• An alert fires when transmit utilisation on any API server exceeds 70% for ten minutes.
• New scheduled jobs on production hosts require a bandwidth and priority setting in code review.
• The tc and nftables rules are applied by configuration management, not by hand.

Security implications

Unexpected sustained outbound traffic is also a classic sign of data exfiltration, so the same tools are part of incident response. Here, verifying the destination, the user and the binary was what allowed us to treat it as an operational problem. The new utilisation alert doubles as an exfiltration signal. Backups themselves are encrypted by restic before upload, so the bandwidth limit does not change their confidentiality.

Prevention checklist

• Keep historical network counters (sar) so nightly problems can be investigated in the morning.
• Rate-limit bulk transfers on hosts that also serve users.
• Schedule batch jobs from measured traffic, not assumptions about time zones.
• Shape traffic by class as a safety net for jobs that forget limits.
• Alert on sustained high outbound utilisation.

References

• sysstat sar manual: https://man7.org/linux/man-pages/man1/sar.1.html
• ss manual: https://man7.org/linux/man-pages/man8/ss.8.html
• tc-htb manual: https://man7.org/linux/man-pages/man8/tc-htb.8.html
• restic documentation, backup options: https://restic.readthedocs.io/en/stable/040_backup.html