Random 5-second delays on outbound calls from Kubernetes pods: the conntrack DNS race

Latency histograms for our checkout service had a strange shape: a normal hump around 80 ms and a small, sharp spike at about 5,080 ms. Roughly 0.5% of requests to the payment provider's API took exactly five seconds longer than they should. Timeouts were set at 10 seconds, so the calls succeeded, but customers noticed the slow page.

Tracing showed the extra five seconds before the HTTP request even started: inside the client's connection setup, during name resolution.

Reproducing it

From a pod on an affected node, repeated lookups exposed the same pattern:

kubectl run dnstest --rm -it --image=nicolaka/netshoot -- bash
for i in $(seq 1 500); do
  /usr/bin/time -f "%e" getent ahosts api.payments-provider.example > /dev/null
done 2>&1 | sort -n | uniq -c | tail -3

    496 0.00
      1 0.01
      3 5.01

Three lookups out of 500 took five seconds. dig, which sends one query at a time, never showed the delay. getent, which uses glibc's resolver like our application, did.

Looking at the kernel

The nodes' conntrack statistics showed insertion failures increasing:

# on the node
sudo conntrack -S | awk '{print $1, $7}' | head -4

cpu=0 insert_failed=1841
cpu=1 insert_failed=1795
cpu=2 insert_failed=1902
cpu=3 insert_failed=1688

insert_failed counts packets dropped because a conntrack entry could not be inserted, typically due to a race between two packets of the same flow. A packet capture on the pod's interface during a slow lookup showed two queries leaving at the same instant (an A and an AAAA query, from the same source port), one reply arriving, and then, five seconds later, a retry.

What we ruled out

• CoreDNS overload. CoreDNS CPU was below 10%, and its metrics showed sub-millisecond response times. The slow lookups never reached it a second time until the retry.
• Upstream DNS. The delay occurred for cluster-internal names too.
• ndots and search domains. Those cause extra queries and steady added latency, covered in an earlier post; here the delay was exactly the timeout, and intermittent.

Root cause

glibc's resolver sends the A and AAAA queries for a name in parallel, from the same UDP socket. The pod reaches CoreDNS through a Service IP, so both packets go through kube-proxy's DNAT rules and conntrack. When the two packets are processed simultaneously on different CPUs, both try to create the same conntrack entry; one wins, and the other is dropped (insert_failed). The application's resolver then waits for the missing reply until its timeout, five seconds by default in glibc, and retries.

Why it is intermittent

The race only happens when both packets are handled at nearly the same moment on different CPUs, which depends on timing and load. On a busy node with many cores it happens a few times per thousand lookups, enough to show up in percentiles but rare enough to be ignored in casual testing.

Why some clients never see it

• Tools like dig send one query at a time.
• musl libc (Alpine images) handles A and AAAA queries differently and is affected in different ways.
• Applications with their own DNS caching or their own resolver (Go's pure-Go resolver, for example) may send queries differently or cache results.
• Clients connecting through an HTTP keep-alive pool resolve names rarely.

Our checkout service was a Python application on a Debian-based image, using glibc, opening new connections frequently.

Confirming with a packet capture

To be sure, we captured DNS traffic on the node for one pod's interface while running the lookup loop:

sudo tcpdump -ni any 'udp port 53 and host 10.20.41.18' -w /tmp/dns.pcap

In the slow cases, the capture showed the pattern clearly: two queries with different IDs (one A, one AAAA) leaving the pod within microseconds of each other from the same source port, one response, and then nothing for five seconds until the resolver retried both. In the fast cases, both responses came back within a millisecond. The missing response had never been answered: it was the second query that conntrack had dropped, so CoreDNS never saw it.

Environment

Short-term: resolver options

glibc has an option that sends the two queries from different sockets, avoiding the same-flow race:

spec:
  dnsConfig:
    options:
      - name: single-request-reopen
      - name: timeout
        value: "1"
      - name: attempts
        value: "3"

single-request-reopen removed most of the five-second delays for glibc clients. Lowering timeout to one second limits the damage of any remaining drop. Neither helps musl-based images, which ignore these options.

Long-term: NodeLocal DNSCache

NodeLocal DNSCache runs a DNS cache as a DaemonSet on every node, listening on a link-local address. Pods send DNS queries to the cache on their own node. Those queries do not go through the kube-dns Service DNAT, so the conntrack race in that path does not happen, and the cache forwards misses to CoreDNS over TCP.

We deployed it with the upstream manifest, setting the link-local IP and the kube-dns ClusterIP:

kubedns=$(kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}')
localdns=169.254.20.10
domain=cluster.local
sed -e "s/__PILLAR__LOCAL__DNS__/$localdns/g" \
    -e "s/__PILLAR__DNS__DOMAIN__/$domain/g" \
    -e "s/__PILLAR__DNS__SERVER__/$kubedns/g" nodelocaldns.yaml | kubectl apply -f -

In iptables mode, node-local-dns intercepts traffic to the kube-dns IP on each node, so pods did not need a new nameserver address in their resolv.conf.

We kept single-request-reopen as well, since it costs nothing.

Alternatives and trade-offs

• Application-level DNS caching reduces lookups, but every language and library does it differently.
• Kernel fixes. Linux kernels since 5.0 include patches that reduce this conntrack race for some cases; our node image's kernel was older. Upgrading nodes helps, but we did not want to rely on kernel version alone.
• Using TCP for DNS (use-vc option) avoids the UDP race but adds latency to every lookup.

How to test the fix

for i in $(seq 1 5000); do
  /usr/bin/time -f "%e" getent ahosts api.payments-provider.example > /dev/null
done 2>&1 | awk '$1 > 1' | wc -l
# 0

Confirming in production

After NodeLocal DNSCache rolled out, the 5-second spike disappeared from the checkout latency histogram. insert_failed counters on the nodes stopped growing at their previous rate, and CoreDNS query volume dropped by about 70% because of the node-level cache.

Regression guard

• A Prometheus alert on the rate of node_nf_conntrack_stat_insert_failed (from node_exporter) per node.
• A synthetic job runs 1,000 glibc lookups per node every hour and records the slowest.
• The node image upgrade plan moves to a current kernel.

Security implications

NodeLocal DNSCache is another component that sees every DNS query from every pod; it runs with host networking and needs the same care as CoreDNS (current version, minimal privileges). It also improves resilience: pods keep resolving cached names during a short CoreDNS disruption.

Prevention checklist

• When latency spikes are a fixed number of seconds, suspect a timeout and retry.
• Check conntrack -S for insert_failed on nodes with DNS issues.
• Run NodeLocal DNSCache on clusters with high DNS volume.
• Set single-request-reopen and a short resolver timeout for glibc workloads.
• Keep node kernels current.

References

• Kubernetes, Using NodeLocal DNSCache: https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/
• Kubernetes, DNS for Services and Pods, pod DNS config: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-dns-config
• resolv.conf manual, single-request-reopen: https://man7.org/linux/man-pages/man5/resolv.conf.5.html
• conntrack-tools: https://conntrack-tools.netfilter.org/