Every second build on our Jenkins agent failed at checkout with Permission denied while deleting node_modules. A docker run step in the pipeline had written files as root into the bind-mounted workspace. How we proved it, fixed the user mapping and made cleanup reliable.
We built a pre-push gate that runs linters, secret and SAST scans, containerised tests and an LLM review of the diff before code leaves a laptop. The hard parts were not the model call: ordering, cost caps, redaction, fail-open rules and keeping CI as the real authority.
Passing --workers 4 to docker run crashed the container with exec: "--workers": executable file not found. Arguments after the image name replace CMD entirely; they do not append. We explain the override rules in Docker, Compose and Kubernetes and the image layout that makes extra flags safe.
A Node.js API image had grown to 1.1 GB, slowing every deploy and autoscaling event. Step by step, we measured where the size came from, moved to a multi-stage build, chose a base image deliberately and cleaned the build context, ending at 96 MB, with the trade-offs of each step.
When a small self-hosted service lost its disk, we restored the MySQL Docker volume from last night's tar backup, and MySQL refused to start with InnoDB corruption errors. The backup had been a file copy of a running database. How we recovered from an older dump, and the backup approach and restore drills we use now.
Connecting WhatsApp, Telegram or Slack to an LLM agent that can run tools on your server is a few hundred lines of glue and a large attack surface. This is the architecture we use: normalised channel adapters, sender pairing, a locked-down tool sandbox and hard budget caps.
Our queue workers took exactly 30 seconds to stop on every deploy and lost the job they were processing. Exit code 137 in every pod. The Dockerfile used shell-form CMD, so /bin/sh was PID 1 and never forwarded SIGTERM to Node. Exec form plus a real shutdown handler fixed it.
During a traffic spike, new GPU pods for our LLM service took nine minutes to become ready, and users waited or got errors. The image contained 15 GB of model weights, so every new node pulled the whole thing. Separating weights from the image and caching them per node brought scale-up under two minutes.
Firewall rules describe what you meant to expose; a scan from outside shows what you actually expose. This is the repeatable nmap routine we run against our own hosts, including the Docker port that slipped past ufw and how we diff results week to week.
Why does deleting a file in a later Dockerfile step not shrink the image? Why is the first write to a large file inside a container slow? A hands-on explanation of image layers and overlayfs, built with mount commands you can run yourself, and what it means for images and databases.
Running the test suite inside our app container dropped every table in the development database. Docker Compose injected DB_DATABASE as a real environment variable, and phpunit.xml's env entries do not override existing variables. Here is the full precedence chain and three layers of defence.
Every Docker build in Jenkins started failing with permission denied on docker.sock after we rebuilt the agent image. The cause was a group ID mismatch, and the fix made us face what that socket really grants.
We enabled BuildKit's GitHub Actions cache and our multi-stage Go build still took 9 minutes on every run. The default cache export mode only stores layers of the final image, not the builder stage where all the time is spent. mode=max and per-branch scopes cut it to 2 minutes.
A worker image built fine but every container exited at once with "no such file or directory" for a file that clearly existed. A Windows checkout had turned the shebang into /bin/bash with a carriage return. We fixed it with .gitattributes and a CI check.
Our React app called https://undefined/v1/... in production while dev worked fine. Vite replaces import.meta.env at build time, and our Docker image set the variable only at runtime. We fixed the build and then moved to runtime config.
After moving our Symfony app to the official php-fpm image, median response time went from 38 ms to 121 ms with no code change. OPcache was never enabled in the new image, and checking it from the CLI hid the problem. Here is how we found it and tuned it.
Our self-hosted LLM box produced three tokens per second on a server with an RTX 4090. ollama ps reported 100% CPU: the container had no access to the GPU. Installing the NVIDIA Container Toolkit and requesting the GPU in Compose brought it to 120 tokens per second.
After rebuilding our inference image, models silently ran on CPU on half the GPU fleet. The new PyTorch wheel targets CUDA 12, and older nodes still had a driver that only supports CUDA 11. We aligned drivers, added a startup check that fails without a GPU, and pinned images.
Our single-host production server stopped accepting writes at 03:12 with no space left on device. Docker had quietly filled the disk with a 31 GB container log, old images and build cache. Here is how we found each piece and stopped it from coming back.
A hotfix image pushed from an M3 laptop crash-looped on our x86 servers with exec format error. The image was linux/arm64 only. We moved builds to CI with buildx multi-platform manifests and stopped laptop pushes to the registry.
Training jobs that ran fine on a workstation crashed in containers with a bus error in DataLoader workers. Workers pass batches through shared memory, and containers get only 64 MB of /dev/shm by default. A memory-backed emptyDir in Kubernetes and --shm-size in Docker fixed it.
Every push to our FastAPI service spent seven minutes reinstalling the same Python packages. COPY . . ran before pip install, and a missing .dockerignore let .git changes bust the cache. Reordering layers and a cache mount cut builds to 50 seconds.