Jenkins agent gets permission denied on /var/run/docker.sock after an image rebuild

On a Monday morning every pipeline that built a container image went red within the first minute. Checkout and unit tests still passed; only the docker build stage died. The console output was the same on all four agents:

+ docker build -t registry.example.com/billing-api:4f1c2d9 .
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Head "http://%2Fvar%2Frun%2Fdocker.sock/_ping": dial unix /var/run/docker.sock: connect: permission denied

Nothing in the Jenkinsfile had changed. The only change over the weekend was a scheduled rebuild of our agent image, which runs the Jenkins inbound agent plus the Docker CLI, and talks to the host daemon through a bind-mounted socket.

Reproducing it

We started a throwaway container from the new agent image on one of the build hosts, with the same mount the agents use:

docker run --rm -it \
  -v /var/run/docker.sock:/var/run/docker.sock \
  ci/jenkins-agent:2025.07.14 bash

jenkins@3b9e1c0f5a21:~$ docker ps
permission denied while trying to connect to the Docker daemon socket ...

The previous tag, ci/jenkins-agent:2025.06.30, worked from the same host with the same command. So the host and the daemon were fine; something inside the image had moved.

What we ruled out

• Daemon down or restarted. systemctl status docker on the host was healthy and docker ps as root on the host worked.
• Socket path changed. The socket was still /var/run/docker.sock; no DOCKER_HOST override was set in the agent environment.
• Jenkins credentials or the Docker Pipeline plugin. The failure happened in a plain sh step before any registry login, and reproduced outside Jenkins in a bare shell.

Root cause

We compared numeric IDs on both sides of the mount:

# on the host
$ stat -c '%U:%G %u:%g %a' /var/run/docker.sock
root:docker 0:988 660

# inside the new agent image
jenkins@3b9e1c0f5a21:~$ id
uid=1000(jenkins) gid=1000(jenkins) groups=1000(jenkins),999(docker)

The socket is owned by group 988 on the host. Inside the container, the jenkins user is in a group called docker, but with GID 999. File permissions are checked by number, not by name, so the kernel saw a process with groups 1000 and 999 trying to open a 0660 socket owned by group 988, and refused.

The old image had pinned groupadd -g 988 docker in the Dockerfile. During a cleanup someone simplified it to a plain groupadd --system docker while switching the CLI install to a COPY --from of the official CLI image. Without -g, groupadd took the next free system GID. The name stayed the same, the number did not.

What is happening under the hood

A bind-mounted Unix socket keeps the host's ownership and mode. When a process in the container calls connect() on it, the kernel checks the caller's UID and supplementary GIDs against the socket inode, exactly as it would for a file. Container user namespaces are not in play here (we run with the default, no userns remap), so UID 1000 in the container is UID 1000 on the host, and GID 999 is GID 999.

Group names live in /etc/group, and each image has its own copy. Two images can both say docker and mean different numbers. Nothing warns you about this; the first sign is a permission error at runtime.

Environment

Minimal reproduction

FROM jenkins/inbound-agent:latest-jdk21
USER root
# Static CLI binary taken from the official image
COPY --from=docker:28.3-cli /usr/local/bin/docker /usr/local/bin/docker
# groupadd without -g picks the next free system GID, here 999
RUN groupadd --system docker && usermod -aG docker jenkins
USER jenkins

Run it with the socket mounted on any host whose docker group is not the same number, and docker ps fails.

Who is affected

Any setup that bind-mounts the host socket into a container running as a non-root user: Jenkins agents started by the Docker cloud plugin, static agents in Docker Compose, GitLab runners using the socket binding pattern, and dev containers. Agents running directly on a VM as a user in the host docker group are not affected by the GID mismatch, but they have a related trap: adding a user to the group does not affect processes that are already running, so the agent service must be restarted after usermod.

The fix

We did two things. First, the immediate unblock: stop relying on a group name baked into the image, and pass the numeric GID of the socket at container start.

 docker run -d --name agent-01 \
   -v /var/run/docker.sock:/var/run/docker.sock \
+  --group-add "$(stat -c %g /var/run/docker.sock)" \
   -e JENKINS_URL=https://ci.example.com \
   ci/jenkins-agent:2025.07.14

--group-add adds a supplementary GID to the container process regardless of whether that GID exists in the image's /etc/group. With the Docker cloud plugin, the same value goes into the template's extra groups field; in Compose it is group_add: ["988"]. We also removed usermod -aG docker jenkins from the Dockerfile so the image no longer pretends to know the host's numbering.

Second, and more important, we treated the outage as a prompt to stop handing the agents the host socket at all. More on that under the security section, because it changed what "fixed" meant for us.

Alternatives and trade-offs

• Pin the GID in the image (groupadd -g 988 docker before installing packages). Works, but couples every image to one host's numbering. A new host built with a different package order breaks it again.
• chmod 666 /var/run/docker.sock. Do not do this. It gives every local user and every container on the host root-equivalent access, and the daemon resets the mode on restart anyway.
• Run the agent as root. Same blast radius as the socket, with less friction. We rejected it.
• Remote BuildKit or rootless builders. Removes the socket from the agent. This is where we ended up for image builds.

How to test the fix

# 1. Socket reachable with the numeric group added
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
  --group-add "$(stat -c %g /var/run/docker.sock)" \
  ci/jenkins-agent:2025.07.14 docker version --format '{{.Server.Version}}'
# expected: 28.3.1

# 2. And still denied without it, proving the image no longer carries a guessed group
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
  ci/jenkins-agent:2025.07.14 docker ps; echo "exit=$?"
# expected: permission denied ... exit=1

Then we replayed the last failed build of three repositories from the Jenkins UI.

Confirming in production

After rolling the agents, we watched the build stage durations and failure counts in the Jenkins metrics plugin for a day. The docker build stage failure rate went from 100% back to its usual background level (flaky registry pushes, mostly). We also added a tiny diagnostic step at the top of the shared library so the next mismatch is obvious in the log:

sh '''
  echo "agent uid/gids: $(id)"
  echo "socket: $(stat -c '%u:%g %a' /var/run/docker.sock 2>/dev/null || echo absent)"
'''

Regression guard

The agent image pipeline now runs a smoke test on a real build host before the tag is promoted: start the container with the production flags and run docker version. If the check fails, the old tag stays live.

Security implications

This is the part that changed our plan. Access to /var/run/docker.sock is equivalent to root on the host. Any job that can run docker can start a privileged container that mounts the host filesystem:

docker run --rm -v /:/host alpine chroot /host id
# uid=0(root) gid=0(root)

On a CI agent, that means any pull request whose pipeline runs on that agent, including a malicious change to a build script or a compromised dependency that runs during the build, can read every secret on the host, tamper with other jobs' workspaces and caches, and persist. The permission error we fixed was, in a sense, the only thing standing between our jobs and root, and we had just removed it.

What we did about it:

1. Image builds moved to rootless BuildKit. A moby/buildkit:rootless daemon runs as a separate service, as UID 1000, without --privileged (it needs seccomp=unconfined and apparmor=unconfined on our hosts, which we accepted for the dedicated builder hosts). Agents talk to it with buildctl over mutual TLS. A job can build and push images but cannot start arbitrary containers or mount host paths.
2. No socket on agents that run untrusted code. Fork pull requests run on a separate agent pool with no Docker access at all.
3. Kaniko is not our replacement. It was the common daemonless answer for years, but the upstream GoogleContainerTools repository was archived in June 2025 and no longer receives fixes. We did not want to adopt an unmaintained builder for a security improvement. Buildah or Podman in rootless mode are reasonable alternatives if BuildKit does not fit.
4. Where the socket must stay (one legacy integration test suite that drives containers), those agents run on single-purpose VMs that hold no other secrets and are rebuilt nightly.

The build command for the BuildKit path looks like this:

buildctl --addr tcp://buildkitd.ci.example.com:1234 \
  --tlscacert /certs/ca.pem --tlscert /certs/cert.pem --tlskey /certs/key.pem \
  build --frontend dockerfile.v0 \
  --local context=. --local dockerfile=. \
  --output type=image,name=registry.example.com/billing-api:${GIT_COMMIT},push=true

Prevention checklist

• Pass the socket's numeric GID with --group-add; never rely on a group name inside an image.
• Smoke-test agent images on a real host before promoting a tag.
• Treat any agent with docker.sock as root on that host and scope its jobs and secrets accordingly.
• Prefer rootless BuildKit, Buildah or Podman for image builds; check a builder is still maintained before adopting it.
• Keep untrusted pull request builds on agents with no container daemon access.

References

• Docker docs, protect the Docker daemon socket: https://docs.docker.com/engine/security/protect-access/
• Docker docs, docker run --group-add: https://docs.docker.com/reference/cli/docker/container/run/
• BuildKit rootless mode: https://github.com/moby/buildkit/blob/master/docs/rootless.md
• Docker docs, rootless mode: https://docs.docker.com/engine/security/rootless/
• Kaniko repository (archived): https://github.com/GoogleContainerTools/kaniko