Istio 503 "upstream connect error" in the first seconds of every pod: app started before the proxy

We rolled out Istio in sidecar mode across the orders namespace to get mTLS and better traffic metrics. Everything worked in steady state. But every deployment now produced a burst of errors in the first few seconds of each new pod's life. Two different symptoms appeared.

The orders API ran database migrations checks at startup, and some pods crashed immediately:

2026-10-09T08:14:02.117Z ERROR startup: cannot connect to postgres orders-db.orders.svc:5432:
dial tcp 10.20.34.12:5432: connect: connection refused

They restarted once and then worked. And upstream callers saw short bursts of 503s from Envoy:

upstream connect error or disconnect/reset before headers. reset reason: connection failure

Timing the startup

The pod's container start times showed the order of events:

kubectl get pod orders-api-6b9f7d-k2xwq -n orders \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.state.running.startedAt}{"\n"}{end}'

api           2026-10-09T08:14:01Z
istio-proxy   2026-10-09T08:14:01Z

Both containers started in the same second, in parallel. The proxy log showed when Envoy actually became ready:

2026-10-09T08:14:02.981Z  info  Envoy proxy is ready

The application tried to reach the database at 08:14:02.117, almost a second before Envoy was ready.

Why connection refused?

With the sidecar injected, an init container sets up iptables rules that redirect all outbound traffic from the pod to Envoy's outbound listener on port 15001. Until Envoy is listening, redirected connections hit a closed port, which returns "connection refused", even though the database itself is fine.

What we ruled out

• Database availability. The database pod was up; connections from already-running pods succeeded.
• mTLS policy mismatch. A PeerAuthentication problem would cause failures continuously, not only at startup.
• Readiness probes on the API. The 503s to callers came during termination and startup of pods, which pointed at proxy lifecycle, not at the app's own readiness logic.

Root cause

Kubernetes starts a pod's regular containers in parallel. The application container ran its startup code before the Envoy sidecar had loaded its configuration and opened its listeners. Outbound connections were redirected by iptables to a proxy that was not listening yet, and failed. A related issue at shutdown caused the callers' 503s: on termination, Envoy could exit before the application finished in-flight requests.

Container start order in a pod

Classic Istio sidecar injection adds istio-proxy as a regular container, so it races the application. Kubernetes native sidecars, stable since 1.33 and enabled by default from 1.29, solve exactly this.

Istio's two answers

• holdApplicationUntilProxyStarts: true makes the injector put istio-proxy first in the container list with a postStart hook that blocks until Envoy is ready. The kubelet starts containers in list order and waits for postStart hooks, so the application starts only after the proxy is ready.
• Native sidecar mode: Istio injects the proxy as a native sidecar init container, which gives correct start and stop ordering from Kubernetes itself. Recent Istio releases support it, enabled by a setting in older versions and on by default in newer ones on supported Kubernetes versions; check your release notes.

Who is affected

Meshed workloads that make outbound connections at startup (database checks, configuration downloads, cache warm-up, service registration), Jobs and CronJobs whose main container may finish before or start before the proxy, and any workload with tight startup or liveness probes. Inbound traffic is usually safe, because readiness gates keep the pod out of Service endpoints until it is ready, but outbound traffic has no such protection.

Environment

Hold the application until the proxy is ready

Mesh-wide, in the Istio configuration:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    defaultConfig:
      holdApplicationUntilProxyStarts: true

Or per workload, with a pod annotation:

metadata:
  annotations:
    proxy.istio.io/config: |
      holdApplicationUntilProxyStarts: true

Existing pods need to be recreated to get the new injection. After a rollout restart, container start times showed the application starting after the proxy's ready time, and the startup crashes stopped.

Moving to native sidecars

On Kubernetes 1.33 with our Istio version, we then enabled native sidecar injection, which also fixes the shutdown order: the proxy is stopped only after the application container exits.

spec:
  values:
    pilot:
      env:
        ENABLE_NATIVE_SIDECARS: "true"

After injection, the proxy appears under initContainers with restartPolicy: Always:

kubectl get pod orders-api-7c1e2f-m9zpd -n orders -o jsonpath='{.spec.initContainers[*].name}'
# istio-init istio-proxy

Graceful shutdown for the 503s

With classic sidecars, the shutdown race also needed EXIT_ON_ZERO_ACTIVE_CONNECTIONS or a terminationDrainDuration, so Envoy kept serving while the application drained. With native sidecars the ordering is handled by Kubernetes. We also added a short preStop sleep in the application so endpoints are removed from load balancers before the process stops accepting connections.

Make the application tolerant anyway

Retrying the database connection at startup with a short backoff is good practice even with correct ordering: databases restart, DNS can blip. Our startup code now retries for up to 30 seconds before exiting.

Alternatives and trade-offs

• Ambient mode removes per-pod sidecars entirely, using node-level ztunnel and optional waypoint proxies, and with them this class of startup race. It is a larger architectural change.
• Excluding the database port from interception (traffic.sidecar.istio.io/excludeOutboundPorts) avoids the race for that one connection but loses mTLS and telemetry for it.

Checking which pods still use classic sidecars

During the transition, a quick query shows which pods have the proxy as a regular container (old injection) and which as a native sidecar:

kubectl get pods -n orders -o json | jq -r '.items[] |
  [.metadata.name,
   (if any(.spec.containers[]; .name=="istio-proxy") then "classic" else "native" end)] | @tsv'

Any classic pod after the restart means a workload has an annotation or template overriding the injection mode.

How to test the fix

kubectl rollout restart deploy/orders-api -n orders
kubectl get pods -n orders -l app=orders-api -w            # no restarts on new pods
kubectl logs -n orders -l app=orders-api --since=5m | grep -c "connection refused"   # 0

Confirming in production

The next ten rollouts produced zero startup restarts and no 503 bursts in the callers' Envoy metrics (istio_requests_total{response_code="503"} stayed flat during rollouts). Pod startup took about one second longer, the time Envoy needs to become ready.

Regression guard

• An alert on container restarts within 60 seconds of pod creation for meshed namespaces.
• A rollout check compares 503 rates during and before each deployment.
• New namespaces get native sidecar injection by default.

Security implications

During the race window, outbound connections failed rather than bypassing the proxy, so no traffic escaped mTLS. That is the safe failure mode. The tempting workaround of excluding ports from interception would have sent database traffic without mesh mTLS; we avoided it.

Prevention checklist

• Remember that regular containers start in parallel; a sidecar can lose the race.
• Enable holdApplicationUntilProxyStarts or native sidecars for meshed workloads.
• Handle shutdown ordering so the proxy outlives in-flight requests.
• Retry dependencies at startup with backoff.
• Watch for restarts and errors in the first seconds after pod creation.

References

• Istio, holdApplicationUntilProxyStarts: https://istio.io/latest/docs/reference/config/istio.mesh.v1alpha1/#ProxyConfig
• Istio, Application requirements and startup ordering: https://istio.io/latest/docs/ops/deployment/application-requirements/
• Kubernetes, Sidecar containers: https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/
• Istio, Ambient mode overview: https://istio.io/latest/docs/ambient/overview/