The CNI Already Moves Your Packets. Calico Is for the Rules.

The first time I put Calico on an EKS cluster, I almost took the whole network down. Not because Calico is fragile, but because I asked it to do a job the cluster was already doing. Once I understood that a managed Kubernetes cluster already has a CNI moving packets, and that Calico was there for something else entirely, the whole thing got simple. This is the mental model I wish I had started with.

Table of contents

The short version

• On EKS the AWS VPC CNI already gives every pod a real, routable VPC IP. There is no overlay to add and no pod routing for you to invent.
• Calico still earns its place, but only for one job: NetworkPolicy. It decides which pod is allowed to talk to which, and the CNI keeps moving the bytes.
• You run Calico in policy-only mode, which means telling it explicitly “do not touch the datapath.” Skip that and Calico fights the CNI for control of the network.
• The native VPC CNI NetworkPolicy exists, but it has no staged mode. Calico’s staged policies let you roll out default-deny by watching first and enforcing later.
• Two network layers in one cluster sounds like too much until you see them doing two different jobs. One routes. One judges.

Two problems that look like one

Every cluster has to answer two questions, and people collapse them into a single word, “networking,” which is where the confusion starts.

The first question is how does a packet get from pod A to pod B. That is the datapath: IP assignment, routing, the actual movement of bytes. The second question is should pod A be allowed to reach pod B at all. That is policy. They feel related because they both live in the network, but they are as different as a road is from a traffic law.

On a self-managed cluster you often solve both with one tool. Classic Calico, for instance, assigns pod IPs and routes them, using BGP or a VXLAN/IPIP overlay to carry pod CIDRs across a network that does not natively know about them. It does the road and the law together. That is a perfectly good model, and for years it was the default.

Managed Kubernetes on a cloud VPC changes the picture, and that is the part worth slowing down on.

The VPC CNI already solved the datapath

On EKS the AWS VPC CNI hands each pod an actual IP address from your VPC, attached through elastic network interfaces on the node. That is a bigger deal than it sounds. It means a pod is a first-class citizen of the VPC, with an address the VPC router already understands.

No overlay. No encapsulation. No pod CIDR that the underlying network has to be tricked into routing. A pod at 10.0.12.34 is reachable the same way an EC2 instance at 10.0.12.34 would be, because to the VPC they are the same kind of thing.

That is the whole reason the “road” problem is already done for you. So when you reach for Calico on EKS, if you let it also try to own IP assignment and routing, you now have two tools both convinced they are in charge of the datapath. That is the near-miss I opened with. The fix is to tell Calico, in no uncertain terms, that it is not doing the datapath.

Policy-only mode, and the flag that matters

Policy-only mode is exactly what it sounds like. Calico enforces NetworkPolicy and nothing else. The VPC CNI keeps doing IP assignment and routing. Two settings make that split explicit:

# Calico Installation (operator), policy-only on top of the AWS VPC CNI
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  cni:
    type: AmazonVPC        # the VPC CNI stays the CNI, Calico does not replace it
  calicoNetwork:
    bgp: Disabled          # Calico is not routing anything, so no BGP

cni.type: AmazonVPC tells the operator that another CNI owns the datapath. bgp: Disabled follows from that: if Calico is not routing pod traffic, it has no reason to speak BGP. Leave BGP on with the VPC CNI in charge and you get two systems trying to program the network. Set both, and Calico quietly steps back to being a policy engine.

There is one prerequisite that bites people. For Calico to map a policy onto a pod, it has to know the pod’s IP, and on EKS you turn that on in the VPC CNI itself:

# enable pod IP annotation on the VPC CNI add-on so Calico can match policy to pods
vpc-cni = {
  configuration_values = {
    env = { ANNOTATE_POD_IP = "true" }
  }
}

And the mirror-image gotcha: do not also enable the VPC CNI’s own native NetworkPolicy. Two policy engines on the same packets is a debugging nightmare where neither kubectl view tells the whole story. Pick one enforcer. I pick Calico, for the reason in the next section.

Why add Calico at all, when the VPC CNI can do policy

The VPC CNI has had native NetworkPolicy support for a while now. It works. So why bolt on another component?

One word: staging. Calico lets you write a policy in staged mode, where it logs what it would have blocked without actually blocking anything. On a live cluster where nobody has ever written a NetworkPolicy, that is the difference between a controlled rollout and a leap of faith. You apply default-deny in staged mode, watch the flows it would have dropped, discover the three connections you forgot existed, fix them, and only then flip to enforce. The native option makes you jump straight to enforce and find out in production.

Here is roughly how the alternatives shook out when I chose:

The honest caveat: Calico OSS filters at L3/L4, meaning IP, CIDR, and port. It does not filter by hostname. If you need “allow egress to api.vendor.com” and that name resolves to a rotating set of addresses, Calico alone will not do it, and you are looking at an egress proxy. Know that limit before you promise a DNS-based rule you cannot deliver.

What a first policy actually looks like

Once the split is clear, the policy itself is boring, which is the goal. Default-deny for a namespace, then explicit allows. Note the DNS allow, because the mistake everyone makes on their first default-deny is forgetting that pods need to reach CoreDNS before they can reach anything by name.

# deny all egress in the namespace, then allow only what is needed
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: app
spec:
  podSelector: {}
  policyTypes: [Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: app
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: kube-system }
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }

One detail I learned the hard way: when you select a namespace, match it with a native Kubernetes label like kubernetes.io/metadata.name, not a label that some controller is supposed to add for you. Native labels are always there. Controller-added labels are there eventually, and “eventually” on a fresh cluster is how a policy silently matches nothing and denies everything. That story deserves its own post, and it is coming.

Final takeaway

The trick with Kubernetes networking on a cloud VPC is refusing to treat it as one problem. The cloud already solved the road: your pods have real VPC addresses and the VPC routes them without any help from you. What is left is the law, deciding who may talk to whom, and that is the only job you should hand to Calico.

Say that out loud in your config. type: AmazonVPC, bgp: Disabled, native policy off, Calico on. Two layers, two jobs, no fight. The CNI moves your packets. Calico decides if they were allowed to move at all.