Argo Rollouts blue-green sent real users to the preview version: an Ingress on the wrong Service

We had moved the storefront API from a standard Deployment to an Argo Rollouts blue-green strategy, so each release could be tested on a preview URL by QA before promotion. The Rollout looked correct, the preview environment worked, and the first two releases were uneventful because QA promoted them quickly.

The third release had a bug in a new price formatting function. QA found it on the preview URL within minutes and paused promotion. At the same time, customer support started receiving screenshots of prices like 1.299,00 on the English site. The broken version was not promoted, yet customers saw it.

Where were requests going?

The storefront pods logged their version on each request. Counting by version in the access logs for the production hostname:

version=4.18.2   requests=41,203   (active, stable)
version=4.19.0   requests=19,876   (preview, NOT promoted)

About a third of production traffic was going to preview pods.

The Rollout and Services

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: storefront
spec:
  replicas: 6
  selector:
    matchLabels:
      app: storefront
  template:
    metadata:
      labels:
        app: storefront
    spec:
      containers:
        - name: api
          image: registry.example.com/storefront:4.19.0
  strategy:
    blueGreen:
      activeService: storefront-active
      previewService: storefront-preview
      autoPromotionEnabled: false

Rollouts manages the storefront-active and storefront-preview Services: it adds a rollouts-pod-template-hash selector to each, pointing at the active and preview ReplicaSets.

kubectl get svc -n shop -l app=storefront -o custom-columns=NAME:.metadata.name,SELECTOR:.spec.selector

NAME                 SELECTOR
storefront           map[app:storefront]
storefront-active    map[app:storefront rollouts-pod-template-hash:6f8d9c7b5]
storefront-preview   map[app:storefront rollouts-pod-template-hash:7a1c2e4f9]

There was a third Service, storefront, with only app: storefront in its selector. It was left over from the Deployment days, and the production Ingress still pointed at it.

kubectl get endpoints storefront -n shop -o jsonpath='{range .subsets[*].addresses[*]}{.targetRef.name}{"\n"}{end}'

The list contained all twelve pods: six stable and six preview.

What we ruled out

• Rollouts promoting early. The Rollout status was Paused with BlueGreenPause, and the active Service still selected the old ReplicaSet hash.
• Caching layers. The CDN does not cache API responses for the storefront; the leak was in the origin.
• A second Ingress. There was one Ingress for the production host, pointing at storefront.

Root cause

The production Ingress routed traffic to a Service whose selector matched every pod with app: storefront. During a blue-green release, the preview ReplicaSet's pods carry the same app label, so that Service load-balanced across both versions. Argo Rollouts only controls the selectors of the Services named in activeService and previewService; it does not know about, or change, any other Service.

How blue-green works in Argo Rollouts

1. A new revision creates a new ReplicaSet (green) while the current one (blue) keeps running.
2. The preview Service selector is switched to green's rollouts-pod-template-hash, so testers can reach green through the preview Service.
3. On promotion, the active Service selector is switched to green. Traffic moves in one step.
4. After scaleDownDelaySeconds, blue is scaled down.

The isolation between blue and green depends entirely on Service selectors containing the rollouts-pod-template-hash. Any Service that selects pods by labels common to both, such as app, bypasses it.

Why the first two releases did not show it

Both had been promoted within about ten minutes of the preview pods starting, and neither had a user-visible difference. Traffic did leak during those windows too; nobody noticed because both versions behaved the same.

Who is affected

Teams migrating from a Deployment to a Rollout who keep their existing Service, Ingress or service mesh routes, and anyone whose internal callers address the workload through a generic Service. The same leak happens with canary strategies without traffic routing, where the stable Service selects all pods and the canary weight is approximated by replica counts; there it is the intended behaviour, which is exactly why blue-green users must not reuse that pattern.

Environment

Immediate

We pointed the Ingress backend at the active Service:

       backend:
         service:
-          name: storefront
+          name: storefront-active
           port:
             number: 80

Preview traffic from the production host stopped within seconds. Then we aborted the rollout of 4.19.0.

kubectl argo rollouts abort storefront -n shop

Remove the ambiguous Service

The old storefront Service was deleted, after checking that nothing else referenced it: we searched the GitOps repository and looked at the ingress controller's upstream configuration and at service-to-service calls in traces. Two internal callers used storefront.shop.svc; they were moved to storefront-active.shop.svc.

Make the preview private

The preview Service now has its own Ingress on an internal hostname, restricted to the office VPN range and authenticated with SSO through the ingress controller's auth integration. QA tests there; customers cannot reach it even by guessing the hostname.

metadata:
  name: storefront-preview
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "198.51.100.0/24"
    nginx.ingress.kubernetes.io/auth-url: "https://sso.internal.example.com/oauth2/auth"
spec:
  rules:
    - host: preview.storefront.internal.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: storefront-preview
                port: { number: 80 }

Analysis before promotion

We also added a pre-promotion analysis that checks error rate and a smoke test against the preview Service, so obvious regressions block promotion automatically:

strategy:
  blueGreen:
    activeService: storefront-active
    previewService: storefront-preview
    autoPromotionEnabled: false
    prePromotionAnalysis:
      templates:
        - templateName: storefront-smoke
      args:
        - name: service
          value: storefront-preview.shop.svc.cluster.local

Alternatives and trade-offs

• Canary with traffic splitting through the ingress controller or a mesh, if gradual exposure is wanted. Blue-green is all or nothing by design.
• Distinct labels per Rollout version: not needed; the pod template hash already distinguishes them, as long as every routing path uses Rollouts-managed Services.

How to test the fix

During a paused blue-green release on staging:

for i in $(seq 1 200); do curl -s https://api.staging.shop.example.com/version; echo; done | sort | uniq -c
# 200 4.18.2   (only the active version)
curl -s https://preview.storefront.internal.example.com/version   # 4.19.0, from VPN only

Confirming in production

Over the next ten releases, production access logs showed only the active version during paused blue-green phases. The preview hostname received traffic only from internal addresses.

Regression guard

• A policy rejects Services in namespaces with Rollouts if their selector matches a Rollout's pods but does not include rollouts-pod-template-hash, unless annotated as intentional.
• A synthetic check during every paused rollout requests the production host 100 times and alerts if any response comes from the preview version.

Security implications

The preview environment exposed unreleased code to the public, which matters beyond bugs: unreleased features, debug settings or incomplete authorization checks would have been reachable. Restricting the preview Ingress to the VPN and SSO, and routing production only through the active Service, closes that path.

Prevention checklist

• Route production traffic only through the Rollout's activeService.
• Delete or fix Services that select pods by shared labels only.
• Put preview endpoints behind network restrictions and authentication.
• Add pre-promotion analysis for automatic checks before promotion.
• Verify during paused releases that production sees only the active version.

References

• Argo Rollouts, Blue Green strategy: https://argo-rollouts.readthedocs.io/en/stable/features/bluegreen/
• Argo Rollouts, Analysis and progressive delivery: https://argo-rollouts.readthedocs.io/en/stable/features/analysis/
• Argo Rollouts, kubectl plugin: https://argo-rollouts.readthedocs.io/en/stable/features/kubectl-plugin/
• ingress-nginx annotations: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/