Kubernetes networking feels like magic until it breaks, then it feels like a crime scene with no witnesses. A request works from one place and not another, nothing logs an error, the packet just does not arrive. The fix is always the same: stop treating the cluster as one thing, and trace the path a packet takes, hop by hop.
Table of contents
Open Table of contents
The short version
- A request to a pod is a chain, not one hop. The packet has to survive every link.
- The pod has a real IP. The service does not. It is a virtual IP that exists only as rules.
- kube-proxy does not carry your traffic. It programs the node so the kernel does the rewriting.
- Most failures are silent: a dropped packet raises no error, it just times out somewhere else.
- Debugging is walking the path in order, asking at each hop: did the packet make it here.
The path, hop by hop
“The pod cannot reach the service” sounds like a single arrow. It is not. It is a sequence of handoffs, each done by a component that knows nothing about the others.
The load balancer picks a node without knowing which nodes hold a healthy pod. The node gets a packet for a ClusterIP that belongs to no real interface, and the kernel’s netfilter chain rewrites that destination (a DNAT) to one real endpoint pod. Then a routing decision: if the pod is local, the packet crosses a veth pair straight into the pod’s network namespace; if not, it takes one more hop through the CNI to another node. Hold that sequence and the question stops being “why is the network broken” and becomes “which hop did the packet not survive.”
The service is a fiction, and kube-proxy just writes it down
The one thing to internalize: a pod has a real IP, a service does not. A service is a couple of API objects (a ClusterIP and a list of endpoints) that kube-proxy turns into rules on every node. That is the whole separation between control plane and data plane.
This is why you cannot usefully ping a service but curl works: the rules rewrite the destination to a real pod first. It is a promise kept by rules, not a machine, so it survives every pod behind it being replaced. And notice what kube-proxy is doing: it watches the objects and writes the rules, then steps out. It never carries a packet. So if you are staring at the kube-proxy pod hunting for a dropped request, you are in the wrong place. The rules live on the node, written one of three ways:
| Mode | How it steers traffic | Where it bites |
|---|---|---|
| iptables | A netfilter rule chain per service | Evaluated in sequence, so huge clusters pay as rules grow |
| IPVS | An in-kernel load balancer with hashing | Scales to many services better, one more subsystem to learn |
| eBPF | Programs in the kernel datapath | Fastest and most flexible, newer, needs a capable stack |
Why the failures are silent
Most cluster network failures produce no error. They produce absence. A rule drops the packet, and something upstream times out with a message pointing nowhere near the cause.
A policy that matches nothing is indistinguishable from a policy that denies everything, right up until you check the selector.
The classic trap: a network policy whose label selector does not match the target. Instead of allowing traffic, it allows nothing, and the default-deny behind it drops everything. I lost an afternoon to exactly this once, a selector referencing a label that was not present yet on a fresh cluster. It read correctly, matched zero pods, permitted zero traffic, no log line. When you select something, verify the selector matches what you think. Do not assume.
Debugging is just walking the path
The path is a known sequence, so debugging is procedure, not inspiration. Bisect it in order:
- Is the pod healthy and listening on that port? Start at the destination.
- Does the service have endpoints? Empty means the selector matches no ready pods.
- Same-node pod can reach it? Splits “app broken” from “network broken.”
- Cross-node pod can reach it? Fails only cross-node points at the cluster network or a policy.
- Is a network policy in the path? Check for default-deny, and whether your allow selects both ends.
Each check eliminates a hop. You are not searching, you are bisecting, and a dropped packet cannot hide from a bisection.
Final takeaway
Kubernetes networking stops being magic once you refuse to treat it as one thing. The pod is real, the service is a fiction held up by rules, kube-proxy wrote those rules but never carries your traffic. Hold the path in your head and a dropped packet is not a mystery. It is a location you have not checked yet.