Post-Mortem: INC-042
Database Connection Pool Exhaustion
Executive Summary: API latency spiked to 4000ms following a traffic surge. An HPA event provisioned 50+ new pods, which instantly exhausted the available connection pool on the primary RDS instance, causing a lockout of downstream services.
OUTAGE ACTIVE: INC-042 (SEV-1)
00:00:00
REAL-TIME METRICS
Ingress Load 6,500 RPS 5x normal baseline (Critical)
API Response Latency 4,200 ms Target threshold: < 100ms
Active DB Connections 100 / 100 Capacity fully exhausted
Active Pods (checkout-api) 10 Pods HPA Target: min 10 / max 50
ACTIVE RUNBOOK REMEDIATIONS
Detection & Response
T-00:00
Traffic Surge Detected
Unanticipated 5x spike in ingress traffic to `/api/v1/checkout`.
nginx-ingress: 1,200 RPS → 6,500 RPS
T+00:02
DB Pool Exhaustion
Critical RDS error: max_connections reached. Latency spiked to 4s.
[ERROR] RDS: remaining connection slots reserved for non-replication superuser connections
The 5 Whys (RCA)
Resolution Report
Root Cause Analysis
The architecture lacked a centralized connection multiplexer. Database stability was directly coupled to application pod counts, causing an immediate failure during aggressive HPA scaling events.
Remediation Applied
Deployed AWS RDS Proxy to handle connection multiplexing, ensuring future scaling events do not overwhelm downstream DB connections.
PODS
PROXY
RDS