Menu

We Killed Our Staging Environment and Moved QA to Production Canaries
📰

We Killed Our Staging Environment and Moved QA to Production Canaries

HackerNoon·Akhil·about 6 hours ago
#M6Tgmp2e
Long article (1881 words) — no AI summary yet.
Reading 0:00
15s threshold

At 11:40 AM on a Monday, nine engineers were sitting in a Slack thread watching a progress bar that had not moved in 45 minutes.

Someone had merged a half-baked Alembic migration into staging-rds-01 and locked the invoices table. Meanwhile, a real 502 bug on /api/v1/exports was actively hitting paying customers in production, and our hotfix could not ship because company policy said every release had to pass a full QA sign-off on staging first.

Four hours later, the hotfix finally went out. Nobody felt safer. Worse, when we checked our incident log that week, we noticed something embarrassing: staging had failed to catch four of our last five production outages anyway.

Why? Because staging had 120 clean seed rows in Postgres, zero concurrent webhook traffic from Stripe, and a Redis instance that never evicted a key.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.