When a Linux server is slow, the order of the first commands matters more than the tools. A ten-minute triage routine that tells CPU, memory, disk and network pressure apart, what each column actually means, and the mistakes that send people down the wrong path.
When a server is slow and the dashboard is vague, a dozen standard commands narrow the problem to CPU, memory, disk, network or a single process in about five minutes. This is the order we run them in and how we read each output.
Every night API latency tripled for an hour and the 1 Gbit uplink of one server sat at 100%. We traced it from interface counters to a host, a socket and finally a backup job with no rate limit, then shaped it with tc and fixed the schedule.
Celery workers stopped picking up thumbnail jobs one by one, with no errors in the logs. A Popen.wait() on a chatty child filled the 64 KiB pipe buffer and both processes waited on each other. We fixed the deadlock, added timeouts, and learned why timeout alone is not enough with shell=True.
A ledger worker went down with No space left on device while du could account for only 31 GB of a 98 GB volume. A log file deleted by logrotate was still open in the JVM and kept growing. We freed it without a restart and fixed the rotation.
An engineer with nine keys in ssh-agent could not log in to a new server: the connection closed with Too many authentication failures before the right key was tried. How OpenSSH offers keys, why MaxAuthTries cuts it off, and a per-host config that fixes it and is safer.
Many runbooks still say to run shred on sensitive files before deleting them. On SSDs, copy-on-write filesystems and cloud volumes, overwriting in place often leaves the old data elsewhere. Why, and the methods that work: encryption, crypto-erase and device sanitize commands.
When a service cannot reach an API, guessing wastes hours. We follow the packet: name resolution, route, TCP handshake, TLS, then HTTP, with one or two commands per layer that each prove or eliminate a whole class of causes.
Uploads and deploys failed with No space left on device while df showed 30% free. The filesystem had run out of inodes: a session directory held 9.6 million tiny files. How we found it with df -i, cleaned it without locking the box, and stopped it coming back.
Our MySQL primary hung and then aborted with the data volume at 100%. Binary logs had grown to 410 GB because a backfill tripled write volume while retention had been raised for a lagging replica. How we recovered, purged safely and stopped it recurring.
Why does deleting a file in a later Dockerfile step not shrink the image? Why is the first write to a large file inside a container slow? A hands-on explanation of image layers and overlayfs, built with mount commands you can run yourself, and what it means for images and databases.
Our nightly report export ran perfectly when we typed it, and failed every night as a systemd service with node: command not found. Services get no login shell, no .bashrc and a working directory of /. Here is how we made the unit self-contained.
A five-minute cron export started taking six minutes as data grew. Two runs then wrote the same CSV at once, and a partner rejected a file with mangled rows. We serialised the job with flock, bounded it with timeout, and published the file with an atomic rename.
Requests on a reporting server took ten seconds every morning while CPU sat mostly idle. iowait at 60% pointed at storage; iostat and pidstat pointed at logrotate compressing a 40 GB log on the same volume as the database. How we found it and moved the work off the hot path.
Every Docker build in Jenkins started failing with permission denied on docker.sock after we rebuilt the agent image. The cause was a group ID mismatch, and the fix made us face what that socket really grants.
During an incident you rarely need a dashboard; you need answers from a log file in under a minute. The handful of grep, awk, sort, uniq, cut and jq patterns we use most, each run against the same sample nginx and JSON logs, with the output and the mistakes to avoid.
Our base playbook reported changed on every host on every run, so nobody trusted the drift report, and a duplicated sysctl line went unnoticed for months. The cause was shell and command tasks that Ansible cannot reason about; modules, creates and changed_when fixed it.
A queue worker with Restart=always was found dead in the morning with result start-limit-hit. systemd had stopped restarting it after five crashes in ten seconds while Redis was briefly down. How to read the failure, tune StartLimit and RestartSec, and stop flapping services from staying dead.
Our thumbnail service hit Errno 24 at about 1,000 concurrent connections even though ulimit -n showed 65535. systemd services never read limits.conf. We set LimitNOFILE in a drop-in and fixed a client leak that was eating descriptors.
Analysts had an explicit rwx ACL on a shared reports directory and still got Permission denied. A routine chmod 750 had quietly lowered the ACL mask, and files from the ETL job were born with a read-only mask. getfacl showed it all in one word: effective.
Our single-host production server stopped accepting writes at 03:12 with no space left on device. Docker had quietly filled the disk with a 31 GB container log, old images and build cache. Here is how we found each piece and stopped it from coming back.
During a launch our nginx reverse proxy returned errors and logged worker_connections are not enough, at well under the expected capacity. Every proxied request holds a client and an upstream connection, and the file descriptor limit was lower still. How we sized it properly.
After a routine deploy our Node.js API failed to start with EADDRINUSE on port 3000. Everyone blamed TIME_WAIT. The real cause was the old node process surviving the restart, because systemd only killed the npm wrapper above it. How we proved it with ss and fixed the unit.
After moving our PHP app to a new Ubuntu 24.04 server, every page returned 502 Bad Gateway. Fixing socket ownership got us online, then traffic spikes brought the 502s back from an undersized FPM pool. Here is how we fixed both.