Default-deny NetworkPolicy shipped, every service lost DNS: allowing egress to kube-dns

Our security roadmap included network segmentation inside the cluster: each namespace denies all traffic by default, then allows only what it needs. We started with the billing namespace, which had five services. The first policy applied was a default deny for both directions:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: billing
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

We had prepared allow policies for service-to-service traffic and planned to apply them right after. In the minute between the two, and after we applied the allow policies too, every billing pod logged errors like:

psycopg.OperationalError: could not translate host name "billing-db.billing.svc.cluster.local" to address:
Temporary failure in name resolution

Get "https://api.payments-provider.example/v1/charges": dial tcp: lookup api.payments-provider.example
on 172.20.0.10:53: read udp 10.20.33.41:52144->172.20.0.10:53: i/o timeout

Even calls to services in the same namespace failed, although we had explicitly allowed them.

Checking from inside the namespace

kubectl run netcheck -n billing --rm -it --image=nicolaka/netshoot -- bash
nslookup billing-db
# ;; connection timed out; no servers could be reached
nc -zv 10.20.34.12 5432      # the database pod IP directly
# Connection to 10.20.34.12 5432 port [tcp/postgresql] succeeded!

Connecting by IP worked; resolving names did not. Our allow policies covered pod-to-pod traffic, but no policy allowed the pods to reach DNS.

What we ruled out

• CoreDNS itself. Pods in other namespaces resolved names normally.
• Kube-proxy or Service routing. Direct pod IPs worked, and the same Service IPs worked from other namespaces.
• The CNI not enforcing policies. The opposite: it enforced them exactly as written.

Root cause

A default-deny egress policy blocks all outgoing traffic from the selected pods, including DNS queries to CoreDNS. CoreDNS runs in kube-system, so DNS traffic leaves the namespace. Without an explicit egress rule to the DNS pods on port 53, both UDP and TCP, every name lookup timed out, which broke every connection made by name, including within the namespace.

How NetworkPolicy evaluates traffic

• Policies select pods with podSelector in their own namespace.
• Once any policy with Egress in policyTypes selects a pod, that pod's egress is denied except what some policy allows. The same applies to ingress.
• Allow rules are additive: the union of all policies that select the pod.
• Rules match peers with podSelector, namespaceSelector, both combined, or ipBlock.

DNS in Kubernetes goes from the pod to the kube-dns Service IP, which kube-proxy translates to CoreDNS pod IPs. NetworkPolicy is evaluated against the pod IPs after translation in most CNIs, so the rule must select the CoreDNS pods, not the Service IP.

The selector detail that bites

To select pods in another namespace, you need a namespaceSelector. Namespaces have an automatic label, kubernetes.io/metadata.name, since Kubernetes 1.21. Combining it with a podSelector in the same peer entry means "these pods in that namespace":

- namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: kube-system
  podSelector:
    matchLabels:
      k8s-app: kube-dns

Writing them as two separate list items (- namespaceSelector: ... and - podSelector: ...) means "any pod in kube-system, or these pods in my own namespace", a common and dangerous mistake.

Why same-namespace calls also failed

It was confusing at first that calls between billing services failed, because we had allowed them. The allow policy was correct: by IP, billing-api could reach billing-db. But applications do not connect by IP. They connect to billing-db.billing.svc.cluster.local, and the first step of every connection is a DNS lookup that leaves the namespace. When that step fails, the client never gets an address to which the allow rule could apply. Any egress default-deny without DNS therefore breaks everything, regardless of how complete the other allow rules are.

Environment

Who is affected

Every cluster whose CNI enforces NetworkPolicy, when a namespace receives an egress default-deny: Calico, Cilium, the Amazon VPC CNI with policy enforcement, and others. Clusters whose CNI does not enforce policies at all accept the objects silently and nothing changes, which is its own trap: the same manifests then break the day enforcement is enabled.

Allow DNS egress for every pod in the namespace

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: billing
spec:
  podSelector: {}
  policyTypes: ["Egress"]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

TCP matters: DNS falls back to TCP for large responses, and NodeLocal DNSCache forwards over TCP. If you run NodeLocal DNSCache, pods talk to a link-local address on the node instead; allow that with an ipBlock for 169.254.20.10/32.

The full baseline per namespace

After this incident, every namespace starts with the same three policies, applied together in one commit:

1. default-deny-all (ingress and egress).
2. allow-dns-egress as above.
3. allow-same-namespace: ingress and egress between pods in the namespace, if the team wants that as a starting point.

Specific policies then add external egress (with ipBlock for provider ranges or an egress gateway) and ingress from the ingress controller's namespace.

Roll out in audit mode first

Our CNI supports a non-enforcing mode for testing in staging; on the cluster itself we now apply new policies to staging namespaces first and compare connection logs before and after. Many CNIs (Cilium, Calico) offer policy audit or flow visibility for exactly this.

Alternatives and trade-offs

• Namespace-wide allow for kube-system. Simpler selectors, but allows egress to everything in kube-system, including the metrics server and controllers.
• Cluster-wide policies. Some CNIs support cluster-scoped policies that add DNS allow rules for every namespace at once. Useful at scale, but CNI-specific.

How to test the fix

kubectl run netcheck -n billing --rm -it --image=nicolaka/netshoot -- bash -c '
  nslookup billing-db && nslookup api.payments-provider.example &&
  dig +tcp billing-db.billing.svc.cluster.local +short &&
  (nc -zv -w 3 198.51.100.10 443 || echo "external egress blocked, as intended")'

Confirming in production

With the DNS policy applied, all billing services resolved names and recovered within seconds. Over the next month we rolled the baseline to 14 more namespaces, each in one commit with all three policies, with no DNS incidents.

Regression guard

• A conformance test in CI deploys the baseline to an ephemeral namespace and runs the netshoot checks above.
• A policy check fails any manifest set that contains a default-deny egress policy without a DNS allow policy for the same namespace.

Security implications

Default-deny is the right goal: it limits lateral movement and data exfiltration from a compromised pod. The fix keeps that while allowing only DNS to the DNS pods. Be aware that DNS itself can be used for data exfiltration; for high-sensitivity namespaces, consider DNS policy at the resolver level (restricting which external domains resolve) in addition to NetworkPolicy.

Prevention checklist

• Ship default-deny egress together with a DNS allow rule, never alone.
• Allow both UDP and TCP on port 53.
• Combine namespaceSelector and podSelector in one peer entry for cross-namespace rules.
• Test policies with a debug pod in the namespace before and after applying.
• Roll out to staging namespaces first and watch flow logs.

References

• Kubernetes, Network Policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
• Kubernetes, Declare Network Policy: https://kubernetes.io/docs/tasks/administer-cluster/declare-network-policy/
• Amazon EKS, Restrict pod network traffic with Kubernetes network policies: https://docs.aws.amazon.com/eks/latest/userguide/cni-network-policy.html
• Kubernetes, automatic namespace label: https://kubernetes.io/docs/reference/labels-annotations-taints/#kubernetes-io-metadata-name