Argo CD started the app before the database migration finished: sync waves and PreSync hooks

Two days after Christmas, with a skeleton crew on call, we released version 3.9 of our invoicing API. The release added a billing_currency column to the invoices table. Within a minute of the Argo CD sync, error rates on invoice creation jumped to 40 percent:

ERROR: column "billing_currency" of relation "invoices" does not exist (SQLSTATE 42703)

Ninety seconds later the errors stopped by themselves. Nobody had touched anything. That pattern, an error burst that heals itself, is almost always an ordering problem.

Reproducing it

The Argo CD application history showed the sync applied 14 resources in a single wave. Timestamps from the cluster made the overlap visible:

kubectl -n invoicing get events --sort-by=.lastTimestamp \
  | grep -E "invoicing-migrate|invoicing-api-7c" | head -8

10:02:11  Normal  SuccessfulCreate  job/invoicing-migrate-3-9-0       Created pod: invoicing-migrate-3-9-0-xk2lp
10:02:11  Normal  ScalingReplicaSet deployment/invoicing-api          Scaled up replica set invoicing-api-7c5f9 to 2
10:02:19  Normal  Pulled            pod/invoicing-migrate-3-9-0-xk2lp  Successfully pulled image "flyway/flyway:10.20"
10:02:24  Normal  Started           pod/invoicing-api-7c5f9-8n4tq      Started container api
10:02:31  Normal  Started           pod/invoicing-migrate-3-9-0-xk2lp  Started container migrate
10:03:44  Normal  Completed         job/invoicing-migrate-3-9-0        Job completed

The new API Pods started at 10:02:24 and became Ready a few seconds later. The migration container did not even start until 10:02:31 (its image was not cached on that node) and the ALTER TABLE waited on a lock behind a long-running report query until 10:03:44. For over a minute, new code served traffic against the old schema.

What we ruled out

A failed migration. Flyway's history table showed version 39 applied successfully.

Old Pods. The error came from Pods with the new image tag. Old Pods did not reference the new column.

Readiness probes. They worked as designed, but /healthz only checks that the HTTP server is up. It does not query the schema, and it should not: a probe that fails on missing columns would also take healthy Pods out of rotation during unrelated database blips.

Root cause

The migration Job was a plain manifest in the Helm chart, named after the image tag so each release got a fresh Job. Argo CD treats it like any other resource. With no sync wave annotations, every resource was in wave 0, and Argo CD applies all resources of a wave at once. Nothing told it that the Deployment depended on the Job. The ordering we had relied on for a year was luck: images were usually cached and migrations usually took two seconds.

What is happening under the hood

An Argo CD sync runs in phases, and each phase in waves:

1. Phases: PreSync, then Sync, then PostSync. A SyncFail phase runs only if the sync fails. Resources without a hook annotation belong to the Sync phase.
2. Waves: inside each phase, resources are grouped by argocd.argoproj.io/sync-wave (default 0, negative values allowed). Argo CD applies the lowest wave, waits until every resource in it is healthy, then moves to the next.
3. Health: a Job is Progressing while running, Healthy when it completes, and Degraded when it fails. A Deployment is healthy when its rollout has finished. A failed or degraded resource in a wave stops the sync there.

Hooks are resources annotated with argocd.argoproj.io/hook. They are created at their phase, are not part of the application's normal diff, and are cleaned up according to argocd.argoproj.io/hook-delete-policy (BeforeHookCreation, HookSucceeded, HookFailed). Helm chart hooks are translated: pre-install and pre-upgrade become PreSync, post-install and post-upgrade become PostSync, and helm.sh/hook-weight acts as the sync wave.

Two consequences matter for migrations. A Job in the same wave as the Deployment gives no ordering at all. And a plain Job with a fixed name fails on the next sync, because a Job's spec.template is immutable:

Job.batch "invoicing-migrate" is invalid: spec.template: Invalid value: core.PodTemplateSpec{...}: field is immutable

That error is why our chart had put the image tag in the Job name in the first place.

Environment

Minimal reproduction

This runs on kind with Argo CD installed from the upstream manifests. It uses a PostgreSQL Pod and two containers that stand in for the migration and the app: the "migration" waits 30 seconds then creates a table, the "app" exits with an error if the table is missing.

kind create cluster --name waves
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.13.2/manifests/install.yaml
kubectl create namespace demo
kubectl -n demo run db --image=postgres:16 --env=POSTGRES_PASSWORD=devpass --port=5432
kubectl -n demo expose pod db --port=5432

Put these manifests in a Git repo and point an Argo CD Application at the directory:

apiVersion: batch/v1
kind: Job
metadata:
  name: migrate-v1
spec:
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: postgres:16
          env:
            - name: PGPASSWORD
              value: devpass
          command: ["sh", "-c", "sleep 30; psql -h db -U postgres -c 'CREATE TABLE IF NOT EXISTS invoices (id serial primary key, billing_currency text)'"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 1
  selector:
    matchLabels: {app: demo}
  template:
    metadata:
      labels: {app: demo}
    spec:
      containers:
        - name: app
          image: postgres:16
          env:
            - name: PGPASSWORD
              value: devpass
          command: ["sh", "-c", "psql -h db -U postgres -c 'SELECT billing_currency FROM invoices LIMIT 1' && sleep infinity"]

Sync it. The app container fails with ERROR: relation "invoices" does not exist and restarts a few times before the Job finishes, exactly our production pattern, compressed.