ALB weighted routing for a zero-downtime API cutover

A payments orchestration API, Node 16 monolith on EC2, PostgreSQL, Redis rate limits, needed Node 20 and OpenSSL 3 compatibility without a maintenance window. Regulatory constraints required auditable rollback within 15 minutes.

Prior big-bang deploys had caused brief 502 spikes during target registration. Leadership mandated canary validation on real traffic fractions.

Problems encountered: sticky sessions on an older mobile client pinned some users to blue for days, mitigated by TTL reduction and forced rebalance after 24h; green instances failed health check until we increased grace period for JVM-adjacent native module warmup; idempotency keys on POST /transfer masked duplicate retries during a 30-second routing flap, validated via reconciliation job.

Architecture: second Auto Scaling group (green) behind existing ALB; weighted target group routing starting 5% / 95%; health checks on /healthz with dependency probes (DB, Redis); deregistration delay set to 120s for connection draining.

Runbook: promote weights 5 → 25 → 50 → 100 over four hours with automatic rollback if 5xx rate exceeds 0.1% for five minutes; CloudWatch alarms on p99 latency and error budget.

Outcome: full cutover with zero failed transactions in settlement reconciliation; p99 latency improved 18% on green due to Node 20 event loop gains. Described as a regulated fintech API partner, name withheld.