Moving production EC2 from public subnet to private NAT, zero downtime

An insurtech policy API ran on a single t3.large EC2 in a public subnet, SSH open to 0.0.0.0/0, application port exposed directly, RDS reachable only via security group referencing that instance's elastic IP. Security audit mandated private subnet placement behind ALB within 30 days, without scheduled downtime.

Target: EC2 in private subnet across two AZs; NAT Gateway for outbound webhooks and third-party APIs; ALB terminates TLS; Session Manager replaces SSH; RDS security group allows only app SG.

Rollback plan: keep public instance stopped but not terminated for 72 hours; Terraform state pinned; runbook tested in game-day drill.

Problems: hardcoded public IP in a partner allowlist, required coordinated whitelist update; Let's Encrypt HTTP-01 challenge broke when we removed direct 443 to instance, moved to DNS-01 via Route 53; background workers called external KYC API with IP-based rate limits, NAT Gateway elastic IP pre-registered with vendor.

Cutover technique: launch parallel instances in private subnet registered to new target group; ALB weighted forward 10% traffic; validate webhooks and cron via SSM port-forward logs; drain public instance connections over 300s; de-register public target; update Route 53 alias to ALB only.

Outcome: zero customer-facing 5xx during migration window; attack surface reduced, no public instance IPs; compliance scan passed next quarterly review. Engagement referenced as digital insurtech API, name withheld.