GitHub Actions ARC runners: "Cannot connect to the Docker daemon" and choosing a container mode

To cut CI costs and get access to private networks, we moved our GitHub Actions jobs from GitHub-hosted runners to self-hosted runners on our EKS cluster, using Actions Runner Controller (ARC) with runner scale sets. Test jobs worked immediately. Every job that built a container image failed:

Run docker build -t registry.example.com/orders-api:${GITHUB_SHA} .
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Error: Process completed with exit code 1.

Jobs that used container: or service containers failed even earlier, in the "Initialize containers" step, because the runner had no container runtime to start them with.

On GitHub-hosted Ubuntu runners, Docker is preinstalled. On ARC runners, it is not.

What the runner pod contains

kubectl get pods -n arc-runners -l actions.github.com/scale-set-name=linux-x64 -o jsonpath='{.items[0].spec.containers[*].name}'
# runner
kubectl exec -n arc-runners <runner-pod> -c runner -- ls -l /var/run/docker.sock
# ls: cannot access '/var/run/docker.sock': No such file or directory

One container, runner, based on the official runner image. No Docker daemon, no socket.

What we ruled out

• Missing docker CLI. The CLI was present in the runner image; only the daemon was missing.
• Node Docker socket. EKS nodes use containerd, not Docker, so mounting a host socket was not even an option, and we would not have wanted it.
• Permissions. There was no socket to have permissions on.

Root cause

ARC runner scale sets run each job in a runner pod with a single container by default. Workflows that call docker directly, use container: jobs or service containers need a container runtime the runner can talk to. That requires choosing and configuring one of ARC's container modes; we had installed the scale set with defaults.

ARC container modes

ARC's gha-runner-scale-set Helm chart supports two container modes, plus the default:

dind gives the closest experience to GitHub-hosted runners: docker build, docker run, service containers, everything works. The cost is a privileged container in every runner pod. A privileged container can, with some effort, escape to the node.

kubernetes mode runs job steps in separate pods without Docker. container: jobs and service containers work, but docker build does not, so image builds need a daemonless builder such as BuildKit in rootless mode, Kaniko or Buildah. It requires a persistent volume for the job workspace shared between the runner pod and job pods.

Environment

dind for image-building repositories

We created two scale sets: one in dind mode for repositories that build images, one in default mode for everything else.

# values-dind.yaml for gha-runner-scale-set
githubConfigUrl: https://github.com/example-org
githubConfigSecret: arc-github-app
runnerScaleSetName: linux-x64-docker
minRunners: 0
maxRunners: 30
containerMode:
  type: dind
template:
  spec:
    nodeSelector:
      workload: ci-runners
    tolerations:
      - key: workload
        value: ci-runners
        effect: NoSchedule
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]
        resources:
          requests: { cpu: "2", memory: 4Gi }

With containerMode.type: dind, the chart adds the dind sidecar, an init container that copies runner externals, the shared /var/run volume and the DOCKER_HOST environment variable. Workflows opt in with the scale set name:

jobs:
  build:
    runs-on: linux-x64-docker
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t registry.example.com/orders-api:${{ github.sha }} .

Isolate the privileged runners

Because dind runs privileged, runner pods must not share nodes with production workloads. We created a dedicated Karpenter NodePool with a taint, so only runner pods tolerate it:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: ci-runners
spec:
  template:
    metadata:
      labels:
        workload: ci-runners
    spec:
      taints:
        - key: workload
          value: ci-runners
          effect: NoSchedule
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
      expireAfter: 24h
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: ci-runners
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 2m

The ci-runners EC2NodeClass uses a separate IAM role with no access to production resources and a security group that can reach only the registry, GitHub and package mirrors. WhenEmpty consolidation avoids killing nodes with running jobs.

Runner pods are ephemeral in ARC scale sets: each pod handles one job and is deleted afterwards, which also limits persistence of anything a malicious job might leave behind.

Long term: daemonless builds

For new pipelines we use BuildKit in rootless mode running as a separate service in the CI namespace, with the buildx remote driver, so the runner itself needs no privileges. Repositories move to the default, unprivileged scale set as they switch.

Alternatives and trade-offs

• kubernetes container mode everywhere. No privileged containers, but every image build must change to a daemonless tool, and the shared workspace volume adds latency and storage setup.
• Mounting the node's container runtime socket. Gives every job root on the node; never do this on shared clusters.
• GitHub-hosted larger runners for image builds. No infrastructure to manage, at a higher per-minute cost and without private network access.

How to test the fix

jobs:
  smoke:
    runs-on: linux-x64-docker
    steps:
      - run: docker info --format '{{.ServerVersion}}'
      - run: docker run --rm alpine:3.20 echo ok

Confirming in production

Image-building jobs ran on the dind scale set; median job start time, including node provisioning when scaled to zero, was 70 seconds, and around 10 seconds when nodes were warm. CI costs dropped by roughly half compared with GitHub-hosted runners for the same workload.

Regression guard

• A policy denies privileged pods everywhere except the arc-runners namespace, and in that namespace only on nodes with the ci-runners taint.
• A weekly report lists repositories still using the dind scale set, to track the move to daemonless builds.

Security implications

Self-hosted runners execute code from pull requests. Combined with privileged dind, that is a path from a malicious pull request to a node. We restrict self-hosted runners to private repositories, require approval for workflows from first-time contributors, isolate runner nodes on their own NodePool, IAM role and network, and keep the nodes short-lived with expireAfter: 24h.

Prevention checklist

• Choose an ARC container mode deliberately; the default has no Docker.
• Use separate scale sets for jobs that need Docker and jobs that do not.
• Isolate privileged runners on dedicated, tainted, short-lived nodes with minimal IAM.
• Prefer daemonless image builds for new pipelines.
• Never mount a node's container runtime socket into job pods.

References

• GitHub Docs, Deploying runner scale sets with Actions Runner Controller: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/deploying-runner-scale-sets-with-actions-runner-controller
• ARC, container modes: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/deploying-runner-scale-sets-with-actions-runner-controller#using-docker-in-docker-or-kubernetes-mode-for-containers
• GitHub Docs, Security hardening for self-hosted runners: https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions#hardening-for-self-hosted-runners
• Karpenter NodePools: https://karpenter.sh/docs/concepts/nodepools/