Tag See all →

#nodelocal-dnscache

1 stories

Banner: random 5-second delays on outbound calls from Kubernetes pods caused by the conntrack DNS race

DevOps & CI/CD Bug Report

Random 5-second delays on outbound calls from Kubernetes pods: the conntrack DNS race

Priya Raghavan · Dec 6, 2024 · 7 min read

About one request in two hundred from our pods took exactly five seconds longer than normal. The delay was in DNS: parallel A and AAAA lookups over UDP raced in the kernel's conntrack table and one reply was dropped, so glibc waited its 5-second timeout. How we proved it and fixed it with NodeLocal DNSCache.