DevOps & CI/CD
Bug Report
Random 5-second delays on outbound calls from Kubernetes pods: the conntrack DNS race
About one request in two hundred from our pods took exactly five seconds longer than normal. The delay was in DNS: parallel A and AAAA lookups over UDP raced in the kernel's conntrack table and one reply was dropped, so glibc waited its 5-second timeout. How we proved it and fixed it with NodeLocal DNSCache.