Kubernetes Job never completes because the sidecar keeps running: fixing it with native sidecars

Our nightly revenue export is a Kubernetes CronJob: one container runs a Python script that reads from a managed PostgreSQL instance and writes Parquet files to object storage. Because the database is only reachable through Cloud SQL Auth Proxy, the Pod also runs the proxy as a second container. The export normally takes about 20 minutes.

One morning the data team reported that yesterday's files existed but the Job was still marked active. Six hours after it started, it flipped to failed and our alerting paged:

$ kubectl -n analytics get jobs
NAME                       STATUS    COMPLETIONS   DURATION   AGE
revenue-export-29213460    Failed    0/1           6h         6h2m

$ kubectl -n analytics describe job revenue-export-29213460 | tail -3
  Warning  DeadlineExceeded  2m  job-controller  Job was active longer than specified deadline

During those six hours the Pod looked like this:

NAME                            READY   STATUS     RESTARTS   AGE
revenue-export-29213460-k7v2d   1/2     NotReady   0          3h41m

Reproducing it

Container statuses made the shape of the problem obvious:

kubectl -n analytics get pod revenue-export-29213460-k7v2d \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.state}{"\n"}{end}'

cloud-sql-proxy   {"running":{"startedAt":"2026-07-24T01:00:07Z"}}
export            {"terminated":{"exitCode":0,"reason":"Completed","finishedAt":"2026-07-24T01:19:52Z"}}

The export container had exited 0 after 20 minutes. The proxy was still running, as proxies do. A Pod is not finished while any of its regular containers runs, and a Job only counts a completion when its Pod reaches Succeeded. It reproduced every single night once we looked.

What we ruled out

The export hanging. The terminated state with exit code 0 and the uploaded files ruled it out.

The CronJob controller. The Job object was created on schedule and its Pod started on time. The controller was doing exactly what we asked.

Our old shutdown hack failing randomly. This was the interesting one. The Pod had a wrapper script that called the proxy's /quitquitquit endpoint at the end of the export. Why had it worked for months? It turned out it had not been working reliably at all: a refactor three weeks earlier changed the container command to call the Python module directly, bypassing the wrapper. Until the 6 hour activeDeadlineSeconds fired, every run had looked "slow" rather than broken, and the CronJob's concurrencyPolicy: Forbid meant nothing overlapped.

Root cause

The proxy was a regular container. Kubernetes has no idea which container in a Pod is "the job" and which is "support", so it waits for all regular containers to exit. Our only mechanism for stopping the proxy was an application-level signal that a later change silently removed. Even when it worked, it depended on the export reaching its last line: an OOM kill or an unhandled exception would skip it and leave the Pod hanging the same way.

What is happening under the hood

Since Kubernetes 1.28 (alpha) there is a first-class answer: native sidecar containers. A sidecar is declared in initContainers with restartPolicy: Always. The feature was beta and enabled by default from 1.29 and reached GA in 1.33. The kubelet treats such a container differently from both regular and init containers:

• It starts in its position among the init containers, but the kubelet does not wait for it to exit. It moves on once the sidecar has started, meaning its startupProbe succeeded or, without a startup probe, its process is running.
• It keeps running for the life of the Pod and is restarted if it crashes, regardless of the Pod's restartPolicy.
• It can have readiness, liveness and startup probes, and its readiness counts toward Pod readiness.
• When all regular containers of a Pod with restartPolicy: Never or OnFailure have terminated, the kubelet stops the sidecars, in reverse order of declaration, and the Pod moves to its final phase. The phase is determined by the regular containers, so a sidecar exiting with 143 on SIGTERM does not turn a successful run into a failure.
• On Pod deletion, regular containers are stopped first and sidecars last, so the proxy outlives the app during shutdown too.

That last pair of properties is exactly what a Job with a helper needs: the helper starts first, is ready before the app connects, and goes away by itself when the app is done.

Environment

Minimal reproduction

This runs on any cluster at 1.29 or newer. First the broken version, with a classic two-container Pod:

apiVersion: batch/v1
kind: Job
metadata:
  name: sidecar-legacy
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 120
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: work
          image: busybox:1.36
          command: ["sh", "-c", "echo working; sleep 5; echo done"]
        - name: helper
          image: busybox:1.36
          command: ["sh", "-c", "trap 'exit 0' TERM; while true; do sleep 1; done"]

kind create cluster --image kindest/node:v1.33.1
kubectl apply -f sidecar-legacy.yaml
kubectl get job sidecar-legacy -w

The Job stays at 0/1 completions and fails after 120 seconds with DeadlineExceeded. The work container finished after five seconds.