docker run arguments replaced the whole command: CMD vs ENTRYPOINT override rules

Our report-rendering service ships as a Docker image running Gunicorn. During a load test, an engineer wanted more workers and did what seemed natural:

docker run --rm -p 8000:8000 registry.example.com/reports/render:2.7.1 --workers 4

The container died immediately:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "--workers": executable file not found in $PATH: unknown.

Without the flag, the same image started fine. A day later the same misunderstanding reached our Kubernetes manifests in a different form, where the container started but ignored the bind address and was unreachable. Two symptoms, one rule.

Reproducing it

The Dockerfile ended like this:

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
USER 10001
CMD ["gunicorn", "render.wsgi:app", "--bind", "0.0.0.0:8000", "--workers", "2"]

Inspecting the image showed what Docker would run:

docker image inspect registry.example.com/reports/render:2.7.1 \
  --format 'Entrypoint={{json .Config.Entrypoint}} Cmd={{json .Config.Cmd}}'

Entrypoint=null Cmd=["gunicorn","render.wsgi:app","--bind","0.0.0.0:8000","--workers","2"]

What we ruled out

PATH inside the image. docker run --rm IMAGE which gunicorn printed /usr/local/bin/gunicorn, and that command itself proves the point: the arguments which gunicorn replaced the CMD and ran as the program.

A broken base image tag. The same failure appeared with the previous release.

Shell versus exec form. The CMD was already in exec (JSON array) form, so there was no /bin/sh -c wrapper involved. That matters for signals, but not for this bug.

Root cause

Anything after the image name in docker run replaces the image's CMD completely. With no ENTRYPOINT, the first replacement word becomes the executable, so Docker tried to exec a program called --workers. Arguments never append to CMD. The Kubernetes variant was the same rule with different field names: someone set command: in the Pod spec to add a flag, which replaced the whole thing, including our --bind 0.0.0.0:8000, so Gunicorn fell back to its default 127.0.0.1:8000 and nothing outside the container could reach it.

What is happening under the hood

An image config has two lists, Entrypoint and Cmd. The runtime builds the process argv as Entrypoint + Cmd. The override rules:

Two further details catch people:

• Setting ENTRYPOINT in a Dockerfile resets an inherited CMD. If the base image has a CMD and your Dockerfile adds an ENTRYPOINT, the base CMD is cleared, as the Dockerfile reference states. Declare CMD again after ENTRYPOINT if you want defaults.
• Shell form disables argument passing. ENTRYPOINT gunicorn ... without brackets becomes /bin/sh -c "gunicorn ...", and the shell ignores the CMD and any extra run arguments. Exec form is required for the pattern below to work.

The Kubernetes naming is the most confusing part: the Pod field command maps to Docker's ENTRYPOINT, and args maps to CMD.

Environment

Minimal reproduction

cat > Dockerfile <<'EOF'
FROM busybox:1.36
CMD ["echo", "default", "args"]
EOF
docker build -t cmd-demo .
docker run --rm cmd-demo
docker run --rm cmd-demo extra

default args
docker: Error response from daemon: ... exec: "extra": executable file not found in $PATH: unknown.

The fix

We split the image into a fixed executable and overridable flags. The program and its defaults live in a small entrypoint script set as ENTRYPOINT, and CMD is left empty, so anything passed at run time is treated as extra flags:

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
USER 10001
ENTRYPOINT ["docker-entrypoint.sh"]
# No CMD: the ENTRYPOINT above also clears the CMD inherited from python:3.12-slim.

#!/bin/sh
set -eu
# Allow running another program, for example "sh" or "python manage.py check".
case "${1:-}" in
  -*|"") set -- gunicorn render.wsgi:app --bind "0.0.0.0:${PORT:-8000}" --workers "${WORKERS:-2}" "$@" ;;
esac
exec "$@"

Now:

• docker run IMAGE runs Gunicorn with two workers.
• docker run IMAGE --workers 4 passes --workers 4 as CMD; the script sees a flag and prefixes the full Gunicorn command, keeping the bind address.
• docker run IMAGE sh gets a shell for debugging, because the first argument is not a flag.

The exec "$@" line is essential: it replaces the shell with Gunicorn so Gunicorn is PID 1 and receives SIGTERM directly. Gunicorn's option parsing lets a later --workers override the earlier default, so defaults and overrides compose cleanly. Keeping defaults in the script rather than in CMD matters: a CMD of ["--workers", "2"] would be thrown away by any run-time argument, including an unrelated one such as --print-config.

In Kubernetes we changed the manifest to use args only:

 containers:
   - name: render
     image: registry.example.com/reports/render:2.8.0
-    command: ["gunicorn", "render.wsgi:app", "--workers", "4"]
+    args: ["--workers", "4"]

Alternatives and trade-offs

• Configuration through environment only. Gunicorn reads GUNICORN_CMD_ARGS, and many servers support env-based config. Simpler than an entrypoint script, but every new knob needs documenting, and argument overrides stay broken for anyone who tries them.
• ENTRYPOINT with the full command, no script. ENTRYPOINT ["gunicorn", "render.wsgi:app"] plus CMD ["--bind", "0.0.0.0:8000"] works, but overriding CMD to add a flag drops the bind address again. Put only true defaults in CMD, or require the bind through config.
• Keep CMD only. Fine for images nobody parametrises. Document that overrides must repeat the whole command.

How to test the fix

We added a small test script to the image build pipeline that exercises each override path:

set -e
img=registry.example.com/reports/render:2.8.0
docker run --rm "$img" --print-config 2>/dev/null | grep -E "^(workers|bind)"
docker run --rm "$img" --workers 4 --print-config 2>/dev/null | grep -E "^workers"
docker run --rm "$img" sh -c 'echo shell-ok'
docker image inspect "$img" --format '{{json .Config.Entrypoint}} {{json .Config.Cmd}}'

Expected output:

bind                              = ['0.0.0.0:8000']
workers                           = 2
workers                           = 4
shell-ok
["docker-entrypoint.sh"] null

--print-config (available since Gunicorn 20.1) prints the resolved settings and exits, which makes it a cheap way to assert what the container would actually run.

Confirming in production

After deploying 2.8.0, we confirmed the process tree and arguments inside a running Pod:

kubectl -n reports exec deploy/render -- cat /proc/1/cmdline | tr '\0' ' '; echo
kubectl -n reports exec deploy/render -- python -c "import socket; print(socket.create_connection(('127.0.0.1', 8000)).getpeername())"

/usr/local/bin/python3.12 /usr/local/bin/gunicorn render.wsgi:app --bind 0.0.0.0:8000 --workers 2 --workers 4
('127.0.0.1', 8000)

The repeated --workers flag is expected; Gunicorn applies the last one, and its startup log reported four workers. PID 1 was Gunicorn itself, not the shell script, so rolling restarts still shut down gracefully. The Service endpoints were reachable from other Pods, and the readiness probe passed on the first attempt.

Regression guard

• The override test script runs in CI on every image build.
• hadolint runs on Dockerfiles; its rule DL3025 flags shell-form CMD and ENTRYPOINT, which would silently break argument passing.
• A policy check in our manifest linting warns when a Pod spec sets command for an image that defines an ENTRYPOINT, nudging people toward args.

Security implications

The entrypoint script runs every argument it is given via exec "$@", which is standard and safe because whoever controls container arguments already controls what the container runs. Do not expand arguments through eval or unquoted variables, which would allow injection if arguments come from user input, for example a job runner that templates them. We kept USER 10001 after the COPY so the script and Gunicorn run as non-root, and the script is owned by root and not writable by that user.

Prevention checklist

• Write CMD and ENTRYPOINT in exec form.
• Put the executable in ENTRYPOINT; keep defaults that must survive overrides out of CMD.
• End entrypoint scripts with exec "$@".
• In Kubernetes, use args to change flags and reserve command for replacing the program.
• Re-declare CMD after ENTRYPOINT when building on a base image that sets one.

References

• Dockerfile reference, CMD and ENTRYPOINT: https://docs.docker.com/reference/dockerfile/
• Docker documentation, docker container run: https://docs.docker.com/reference/cli/docker/container/run/
• Kubernetes documentation, Define a Command and Arguments for a Container: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/
• Compose file reference, services: https://docs.docker.com/reference/compose-file/services/
• Gunicorn documentation, Settings: https://docs.gunicorn.org/en/stable/settings.html