We rolled out a default-deny egress NetworkPolicy to a namespace and every pod immediately failed to resolve names, including other services in the same namespace. Policies select pods, DNS runs in kube-system, and the namespace selector needed the right label. The policy set we use now and how we test it.
Calls from our pods to a payment API had 40 to 200 ms of unexplained latency and CoreDNS was the busiest workload in the cluster. Every lookup of api.payments.example went through five search domains first because of ndots:5. FQDNs with a trailing dot and a lower ndots fixed both.
About one request in two hundred from our pods took exactly five seconds longer than normal. The delay was in DNS: parallel A and AAAA lookups over UDP raced in the kernel's conntrack table and one reply was dropped, so glibc waited its 5-second timeout. How we proved it and fixed it with NodeLocal DNSCache.