It Is Always DNS. This Time It Was Hiding in the Defaults.

The joke writes itself. Production started failing in that vague, intermittent way that makes people distrust their own eyes, and the punchline, as it almost always is, was DNS.

But “it is always DNS” is not a diagnosis. It is a genre. The interesting part is never that DNS broke. It is why. And this time the why was not a bad record, a wrong upstream, or a typo in a zone file. It was a default that nobody set on purpose, quietly multiplying every lookup on the cluster until CoreDNS fell over.

Table of contents

The short version

• Name resolution failed intermittently across the whole cluster, not one app.
• CPU, memory, pods and nodes were all healthy. CoreDNS CPU was not.
• The ndots:5 default turned a single external lookup into five DNS queries, four of them useless.
• Multiply that by every pod doing external calls and CoreDNS saturates.
• The fix is a node-local cache plus tuning ndots for the workloads that do not need it.

The symptom nobody could pin down

The reports were annoyingly vague. A job would fail to reach an external API, get retried, and succeed. A service would throw a connection error that pointed at a host that was definitely up. Timeouts appeared in places with no business timing out. Application logs were full of the same unhelpful shape:

getaddrinfo EAI_AGAIN api.external.com
dial tcp: lookup api.external.com: i/o timeout

It was intermittent, it was everywhere, and it followed no pattern anyone could see from inside a single application. That combination is the fingerprint of a shared dependency in trouble.

It was not the app, and it was not the network

The first pass ruled things out fast, because in Kubernetes the reflex is to blame capacity and the reflex is usually wrong.

CPU on the pods?        fine
Memory pressure?        no
Nodes healthy?          yes
Security groups / NACLs? unchanged
The external API?        up, responding, not rate-limiting us

Everything that normally breaks was fine. When the individual pieces are all healthy but the system misbehaves, the problem is almost always in something shared that none of the pieces own. In a cluster, the most shared thing of all, the thing every single pod touches before it does anything useful, is DNS.

CoreDNS was the one thing on fire

Pulling up the CoreDNS pods told the story in one glance. CPU was pegged. Query rate was absurd, far higher than the actual traffic could explain. The processes were healthy in the sense that they were running, and completely underwater in the sense that they could not answer fast enough. Queries were queuing, and queued queries eventually time out, which is exactly the EAI_AGAIN the apps were screaming about.

So the question stopped being “why is DNS broken” and became “why is CoreDNS getting far more queries than we are actually making.”

The app was healthy. DNS was drowning. Those are not the same layer, and the second one never showed up on the dashboards people were watching.

Where the flood came from: ndots

Here is the mechanism, and it is a great example of a sane-looking default doing quiet damage at scale.

Inside a pod, /etc/resolv.conf looks roughly like this:

search my-namespace.svc.cluster.local svc.cluster.local cluster.local region.compute.internal
nameserver 10.96.0.10
options ndots:5

Those search domains exist so that in-cluster names resolve nicely. You can say my-service and the resolver tries my-service.my-namespace.svc.cluster.local for you. Convenient.

The ndots:5 option is the trap. It means: if the name you are looking up has fewer than five dots, treat it as partial and try appending each search domain first, before trying the name as-is.

Count the dots in api.external.com. Three. Fewer than five. So for a fully external, fully qualified name that has nothing to do with the cluster, the resolver dutifully tries all of this first:

api.external.com.my-namespace.svc.cluster.local   -> NXDOMAIN
api.external.com.svc.cluster.local                -> NXDOMAIN
api.external.com.cluster.local                    -> NXDOMAIN
api.external.com.region.compute.internal          -> NXDOMAIN
api.external.com                                   -> answer, finally

One lookup your code asked for. Five queries CoreDNS had to answer, four of them guaranteed garbage. And because both A and AAAA records get requested, in practice it is often double that.

Multiply by the whole cluster

A five-times amplification on one lookup is a curiosity. A five-times amplification on every external lookup, from every pod, in a workload that talks to external APIs constantly, is a query storm. The cluster was effectively running a small denial of service against its own DNS, politely, using nothing but default settings.

Nobody chose this. ndots:5 is the Kubernetes default. It ships that way for good reasons around in-cluster service discovery. It just happens to punish workloads that mostly call outward, and it does so invisibly until CoreDNS runs out of CPU and starts dropping the very queries that would have succeeded.

The fixes, in order of leverage

There is no single switch. There are a few, and the right answer is usually more than one of them.

The highest-leverage change is a node-local DNS cache. You run a small caching resolver on every node, and pods talk to that first. It absorbs the repeated NXDOMAIN answers through negative caching, serves hot names locally, and only forwards real misses to CoreDNS. Central query volume drops hard, and you also remove a class of conntrack and UDP timeout problems that come from every pod reaching across the network for DNS.

The second lever is tuning ndots per workload. A service that only ever calls external, fully qualified names does not need ndots:5. You can lower it for that workload so the resolver stops guessing:

# pod spec: stop expanding fully qualified external names
dnsConfig:
  options:
    - name: ndots
      value: "1"

The third is the cheapest and the ugliest: for a hot external hostname, give it the trailing dot so it is treated as already absolute and skips the search list entirely.

api.external.com.

That final dot is not a typo. It means “this is the fully qualified name, do not append anything.” It looks odd in code and it works, though I reach for it only for specific hot paths, not as a general habit.

There is no free option here. There is the combination whose failure modes you understand, which for us was the node-local cache first, then ndots tuning on the outward-heavy services.

What I would watch after this

DNS deserves its own signals, not just the pod-level metrics that told us nothing. The ones worth graphing:

coredns_dns_requests_total
coredns_dns_responses_total{rcode="NXDOMAIN"}
coredns_cache_hits_total
coredns_cache_misses_total
coredns CPU per pod
node-local cache hit ratio

Final takeaway

Nothing here was exotic. No bad record, no DNS poisoning, no clever failure. Just a default that is correct for the common case and quietly hostile to a workload that does not fit it, scaled up until the shared resolver could not keep up.

That is the actual lesson, and it is bigger than DNS. Cloud and Kubernetes give you sensible defaults so you do not have to think about the fundamentals on day one. They do not repeal the fundamentals. Under the abstraction, resolution still costs queries, queries still cost CPU, and a number as small as ndots still decides whether your cluster hums or drowns.

So yes, it is always DNS. But when someone says that with a shrug, the useful follow-up is never “of course it is.” It is “which default did we forget we accepted.”