Jenkins build fails cleaning the workspace: Docker steps left root-owned files behind

Our frontend pipeline on Jenkins started failing on alternate builds. The first build after a fresh agent passed; the next one failed at the very first step:

[Pipeline] checkout
The recommended git tool is: git
...
ERROR: Error cleaning up workspace
java.nio.file.AccessDeniedException: /var/lib/jenkins/workspace/web-frontend/node_modules/.cache/babel-loader/3f/9a1c.json
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
	...
Also:   hudson.remoting.ProxyException: java.io.IOException: Unable to delete '/var/lib/jenkins/workspace/web-frontend/node_modules'.

Someone on the team had been "fixing" it by SSHing to the agent and running sudo rm -rf on the workspace, which made the next build pass and the one after fail again.

Looking at the files

sudo ls -ln /var/lib/jenkins/workspace/web-frontend | head

drwxr-xr-x  5  996  994  4096 Sep 17 14:02 src
-rw-r--r--  1  996  994   812 Sep 17 14:02 package.json
drwxr-xr-x 412   0    0 16384 Sep 17 14:05 node_modules
drwxr-xr-x  3   0    0  4096 Sep 17 14:06 dist

src and package.json, written by the Git checkout, belonged to UID 996, the jenkins user. node_modules and dist belonged to UID 0, root.

The pipeline step that wrote them

stage('Build') {
  steps {
    sh '''
      docker run --rm -v "$WORKSPACE":/app -w /app node:22-bookworm \
        sh -c "npm ci && npm run build"
    '''
  }
}

The node:22-bookworm image runs as root by default. With a bind mount, the container writes directly into the host directory, and files keep the UID of the process that created them: 0.

What we ruled out

• The Jenkins agent running as root. The agent process ran as jenkins (UID 996), which is why it could not delete root's files.
• A Git issue. Fresh clones worked; only the cleanup of previous build output failed.
• Disk or inode exhaustion. df -h and df -i were normal.

Root cause

A docker run step bind-mounted the workspace and ran as root inside the container. The files it created on the host were owned by root. On the next build, the Jenkins agent, running as an unprivileged user, tried to clean the workspace and could not delete them.

Bind mounts do not translate users

A bind mount exposes a host directory inside the container as-is. There is no ownership mapping in a standard Docker setup: UID 0 in the container is UID 0 on the host, and UID 1000 in the container is whatever user has UID 1000 on the host, if any. Files are written with the numeric UID and GID of the container process.

Rootless Docker and user namespace remapping (userns-remap) change this mapping, but our agents used the standard daemon.

Why the Jenkins Docker pipeline plugin hides the problem

When you use the Docker Pipeline plugin's docker.image(...).inside {} or an agent with docker { }, the plugin runs the container with -u <uid>:<gid> of the Jenkins agent automatically, so files end up owned by the agent user. Our pipeline called docker run from a plain sh step, which gets none of that.

Why it failed only on every second build

The pattern of alternating failures was a useful clue once we understood it. On a clean agent, the checkout step had nothing to delete, so it succeeded, and the build then created root-owned node_modules and dist directories. The next build tried to clean the workspace first and failed on those directories, before any container ran. When someone deleted the workspace by hand with sudo, the cycle started again. Intermittent failures that alternate like this usually mean one run leaves state that the next run cannot handle, so look at what the previous run wrote, not only at the failing run.

Environment

Option 1: run the container as the agent user

The smallest change is to pass the agent's UID and GID to docker run:

 sh '''
-  docker run --rm -v "$WORKSPACE":/app -w /app node:22-bookworm \
+  docker run --rm -u "$(id -u):$(id -g)" \
+    -e HOME=/tmp -e npm_config_cache=/tmp/.npm \
+    -v "$WORKSPACE":/app -w /app node:22-bookworm \
     sh -c "npm ci && npm run build"
 '''

The HOME and npm cache variables matter: UID 996 has no home directory inside the image, and npm otherwise fails trying to write its cache to /.npm.

Option 2: let the plugin manage the container

The cleaner long-term version uses the Docker Pipeline plugin, which mounts the workspace and sets the user for you:

pipeline {
  agent {
    docker {
      image 'node:22-bookworm'
      args '-e HOME=/tmp -e npm_config_cache=/tmp/.npm'
      reuseNode true
    }
  }
  stages {
    stage('Build') {
      steps {
        sh 'npm ci'
        sh 'npm run build'
      }
    }
  }
  post {
    always { cleanWs(deleteDirs: true, notFailBuild: true) }
  }
}

We moved all pipelines to this form. It is shorter, and every sh step inside runs in the container as the agent user.

Clean up the files that already exist

The agent user cannot delete root-owned files, but a container running as root can, without anyone needing sudo on the agent:

stage('Fix legacy ownership') {
  steps {
    sh 'docker run --rm -v "$WORKSPACE":/w alpine:3.20 chown -R "$(id -u):$(id -g)" /w'
  }
}

We ran it once on every agent's workspaces, then removed the stage.

Alternatives and trade-offs

• Rootless Docker or userns-remap maps container root to an unprivileged host UID, which fixes ownership and improves security. It changes how existing volumes and privileged builds behave, so it is a bigger migration.
• Build images that create and use a non-root user (USER node in a custom image). Good practice, but the UID inside still has to match the agent's UID to avoid the same problem in reverse.
• Running the Jenkins agent as root. It would hide the symptom and turn every pipeline into root on the agent host. Not an option.

How to test the fix

# run two builds in a row, then check ownership
find /var/lib/jenkins/workspace/web-frontend ! -uid 996 | head
# no output expected

Confirming in production

After switching to the plugin form, fifty consecutive builds of the frontend pipeline passed, including workspace cleanup. find ... ! -uid 996 on the agents returned nothing.

Regression guard

• A shared library step dockerRun wraps docker run and always adds -u $(id -u):$(id -g); direct docker run calls in sh steps are flagged in review.
• A nightly job on each agent reports any file in workspaces not owned by the agent user.

Security implications

The underlying issue is that the agent user is in the docker group, which is effectively root on the host: any pipeline can start a privileged container and mount /. Fixing ownership does not change that. We documented the risk, restricted which repositories can run on these agents, and started moving container builds to ephemeral agents that are destroyed after each build. The socket permission problem itself, when agents cannot reach Docker at all, is covered in post 295.

Prevention checklist

• Run containers with -u $(id -u):$(id -g) when they write into bind-mounted workspaces.
• Prefer the Docker Pipeline plugin's docker agent or inside over raw docker run.
• Set HOME and tool cache paths when running as a UID without a home directory.
• Use cleanWs in post { always } so failures surface early.
• Treat membership in the docker group as root access.

References

• Jenkins, Using Docker with Pipeline: https://www.jenkins.io/doc/book/pipeline/docker/
• Docker, Bind mounts: https://docs.docker.com/engine/storage/bind-mounts/
• Docker, Isolate containers with a user namespace: https://docs.docker.com/engine/security/userns-remap/
• Jenkins Workspace Cleanup plugin: https://plugins.jenkins.io/ws-cleanup/