Helm production values silently wiped our default env vars: lists replace, maps merge

A small change went to production on a Tuesday: the payments team added a feature flag as an environment variable in values-prod.yaml. Argo CD synced, the new pods started, and within seconds they crashed:

Error: DATABASE_HOST is not set
    at loadConfig (/app/dist/config.js:18:11)

DATABASE_HOST had never been in the production values file. It lived in the chart's default values.yaml, along with five other variables, and had always been there.

The values files

# chart/values.yaml (defaults)
env:
  - name: DATABASE_HOST
    value: payments-db.payments.svc.cluster.local
  - name: DATABASE_NAME
    value: payments
  - name: LOG_FORMAT
    value: json
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://otel-collector.observability:4317
  - name: HTTP_PORT
    value: "8080"
  - name: SHUTDOWN_GRACE_SECONDS
    value: "25"

# values-prod.yaml (before the change)
replicaCount: 6
resources:
  requests: { cpu: 500m, memory: 512Mi }

# values-prod.yaml (after the change)
replicaCount: 6
resources:
  requests: { cpu: 500m, memory: 512Mi }
env:
  - name: FEATURE_INSTANT_REFUNDS
    value: "true"

Rendering both versions

helm template payments ./chart -f values-prod.yaml > after.yaml
git stash && helm template payments ./chart -f values-prod.yaml > before.yaml && git stash pop
diff <(yq '. | select(.kind=="Deployment") | .spec.template.spec.containers[0].env' before.yaml) \
     <(yq '. | select(.kind=="Deployment") | .spec.template.spec.containers[0].env' after.yaml)

< - name: DATABASE_HOST
<   value: payments-db.payments.svc.cluster.local
< - name: DATABASE_NAME
<   value: payments
< ... (four more)
> - name: FEATURE_INSTANT_REFUNDS
>   value: "true"

The rendered Deployment had exactly one environment variable.

What we ruled out

• Argo CD or a stale cache. The rendered output from helm template locally showed the same result.
• Template logic. The template simply ranged over .Values.env; there was no conditional.
• Multiple values files in the wrong order. Only the chart defaults and values-prod.yaml were used.

Root cause

When Helm combines values from the chart's values.yaml and user-supplied files, it deep-merges maps key by key, but a list is a single value: a list in a later file replaces the list in an earlier one entirely. Defining env in values-prod.yaml replaced all six default variables with one.

Helm's coalescing rules

This is deliberate. A generic merge of lists has no right answer (append? merge by index? by some key?), so Helm does not guess. Kubernetes manifests are full of lists, such as env, volumes, tolerations, ports and args, so chart authors who expose them directly as lists hand this trap to every consumer.

Why reviews missed it

The pull request diff showed three added lines in a values file and looked harmless. Nobody reviewed the rendered manifests, which would have shown six variables disappearing. Values-file diffs describe intent; rendered-manifest diffs describe effect.

Who is affected

Any chart that exposes a Kubernetes list directly as a value and is consumed with per-environment override files: env, envFrom, volumes, volumeMounts, tolerations, imagePullSecrets, args, hostAliases and ingress hosts are the usual suspects. Public charts often have the same design, so the issue is not limited to in-house charts. The same rule applies to --set with list indexes and to Argo CD helm.values blocks, which are merged by Helm in the same way.

Environment

Immediate

We reverted the commit; Argo CD synced the previous revision and pods recovered. Then we fixed the chart.

Restructure values as maps

The chart now accepts environment variables as a map, which Helm merges, and converts it to a list in the template:

# chart/values.yaml
env:
  DATABASE_HOST: payments-db.payments.svc.cluster.local
  DATABASE_NAME: payments
  LOG_FORMAT: json
  OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector.observability:4317
  HTTP_PORT: "8080"
  SHUTDOWN_GRACE_SECONDS: "25"

# chart/templates/deployment.yaml (excerpt)
          env:
            {{- range $name, $value := .Values.env }}
            {{- if ne $value nil }}
            - name: {{ $name }}
              value: {{ $value | quote }}
            {{- end }}
            {{- end }}
            {{- with .Values.extraEnv }}
            {{- toYaml . | nindent 12 }}
            {{- end }}

Production now adds a key without touching the others:

# values-prod.yaml
env:
  FEATURE_INSTANT_REFUNDS: "true"

range over a map iterates in sorted key order, so the rendered output is stable between runs, which keeps GitOps diffs clean. Setting a key to null in an override removes it, which is now documented as the way to drop a default.

extraEnv remains a list for the cases maps cannot express, such as valueFrom secret references, and its documentation says clearly that it is not merged.

Apply the same pattern elsewhere

We audited the chart for other lists that consumers override per environment and converted the common ones:

Review rendered manifests, not only values

The more important process fix: every pull request that changes chart values now gets a comment with the diff of rendered manifests per environment, produced in CI:

for env in staging prod; do
  helm template payments ./chart -f values-$env.yaml > /tmp/new-$env.yaml
  git show origin/main:deploy/values-$env.yaml > /tmp/old-values.yaml
  helm template payments ./chart -f /tmp/old-values.yaml > /tmp/old-$env.yaml
  dyff between /tmp/old-$env.yaml /tmp/new-$env.yaml
done

Argo CD's own diff view shows the same thing before sync, if someone looks; making it part of the PR means someone does.

Alternatives and trade-offs

• Repeat the full list in every environment file. Works, but every default change must be copied to every file, and someone will forget.
• Kustomize patches on top of Helm output. Strategic merge patches merge lists like env by name. It adds a second tool to the pipeline.
• Global plus environment lists merged in the template (concat .Values.env .Values.envOverrides). Possible, but duplicates are not deduplicated by name unless you write that logic.

How to test the fix

helm template payments ./chart -f values-prod.yaml \
  | yq 'select(.kind=="Deployment") | .spec.template.spec.containers[0].env[].name'
# DATABASE_HOST DATABASE_NAME FEATURE_INSTANT_REFUNDS HTTP_PORT LOG_FORMAT OTEL_... SHUTDOWN_...
helm unittest ./chart

We added a helm-unittest test asserting that an environment file with one env key still renders all defaults.

Confirming in production

The feature flag shipped the next day with the restructured chart. The rendered manifest diff in the pull request showed exactly one added variable, and the pods started normally.

Regression guard

• Rendered manifest diffs on every values or chart change, posted to the pull request.
• helm-unittest tests for merge behaviour of commonly overridden values.
• A chart lint rule flags new top-level list values that are likely to be overridden per environment.

Security implications

Lists replaced by overrides can drop security settings too: securityContext capabilities lists, NetworkPolicy rules exposed as values, or args that enable TLS. A rendered-manifest diff makes such removals visible. Secrets should continue to come from secret references, not plain values, regardless of structure.

Prevention checklist

• Remember that Helm merges maps and replaces lists.
• Expose frequently overridden collections, such as environment variables, as maps in charts.
• Review rendered manifest diffs, not only values diffs.
• Test merge behaviour with helm-unittest.
• Document how to remove a default (set it to null).

References

• Helm, Values files: https://helm.sh/docs/chart_template_guide/values_files/
• Helm, Chart best practices for values: https://helm.sh/docs/chart_best_practices/values/
• helm template command: https://helm.sh/docs/helm/helm_template/
• Argo CD, Helm: https://argo-cd.readthedocs.io/en/stable/user-guide/helm/