Post-Mortem: INC-108

EKS Cluster Complete Outage (us-east-1)

A deeply technical post-mortem detailing a cascading node failure caused by CNI overlay flooding and ARP cache exhaustion during a massive scaling event.

RESOLVED SEVERITY: SEV-1 MTTR: 42M
Date of Incident
Oct 14, 2025 · 14:02 UTC
Total Duration
42 Minutes
Impact Radius
100% Request Failure
Lead Responder
Vishal Gunjal

Executive Summary: At 14:02 UTC, a sudden Horizontal Pod Autoscaler (HPA) scale-out event triggered the provisioning of 800+ new pods across the EKS cluster within 60 seconds. This massive influx of new IP endpoints flooded the Kubernetes CNI overlay network, instantly exhausting the Linux kernel ARP tables on the underlying EC2 worker nodes. The nodes became unreachable, triggering a complete, cascading cluster outage.

Timeline of Events

T-00:00 (14:02 UTC)
Massive Scale-Out Triggered

An external marketing event drove a 10x traffic spike. The HPA correctly identified the CPU load and requested 800+ new pods to be scheduled immediately across the `us-east-1` node groups.

kubectl describe hpa
Normal SuccessfulRescale 14:02:10 horizontal-pod-autoscaler New size: 850; reason: cpu resource utilization (percentage) above target
T+00:05 (14:07 UTC)
The Kernel Panic (Node NotReady)

As the CNI (Amazon VPC CNI) attempted to allocate secondary IP addresses for the 800+ new pods, the sheer volume of intra-node communication requests overwhelmed the Linux kernel's ARP cache on the EC2 instances. Nodes started dropping off the cluster.

/var/log/messages (dmesg)
[ 3412.019233] IPv4: arp_cache: neighbor table overflow!
[ 3412.021115] IPv4: arp_cache: neighbor table overflow!
[ 3415.119200] kubelet[4512]: E1014 14:07:22.123 PLEG: node status update failed
T+00:42 (14:44 UTC)
Sysctl Tuning & Recovery

The SRE team identified the ARP exhaustion. We rapidly applied a daemonset to inject new `sysctl` kernel tunables across the fleet, drastically increasing the garbage collection thresholds for the neighbor table.

Architecture & Remediation

Architecture Topology Failure
The Root Cause (ARP Table Flood)
By default, the Linux kernel sets `gc_thresh3` (maximum number of ARP entries) to 1024. During our massive HPA scale-out, the CNI provisioned thousands of ENI IPs. The intra-cluster routing caused the nodes to instantly exceed the 1024 limit, causing the kernel to drop all new network packets—effectively severing the node from the control plane.
EC2 Worker Node (us-east-1a) Linux ARP Table Limit: 1024 Status: OVERFLOW Pod 1 Pod 2 Pod ... Pod 800
Action Items / Remediation
We permanently resolved this class of failure by injecting the following sysctl kernel tunables directly into the Terraform EKS Launch Templates, ensuring all future nodes spawn with high-capacity ARP tables.
sysctl — tuning
# Increase IPv4 neighbor table thresholds
sysctl -w net.ipv4.neigh.default.gc_thresh1=1024
sysctl -w net.ipv4.neigh.default.gc_thresh2=4096
sysctl -w net.ipv4.neigh.default.gc_thresh3=8192
Secondary Action Items
  • Implement NodeLocal DNSCache to reduce overall DNS lookup burden on the CNI network.
  • Create Grafana alerts mapping specifically to the `node_arp_entries` metric to detect overflow before it occurs.