Menu

1 / 2

Postmortem: How a Terraform 1.10 Bug Deleted Our Entire Staging Environment 2 Days Before Launch

DEV Community·ANKUSH CHOUDHARY JOHAL·6 months ago
#ViaJyb1t
#code#terraform#postmortem#deleted#state
Reading 0:00
15s threshold

At 14:17 UTC on October 15, 2024, a single terraform apply run on Terraform 1.10.0 destroyed 94% of our staging environment’s infrastructure 48 hours before our Series B launch demo. We lost 12 RDS instances, 47 ECS services, 3 Redis clusters, and 112 S3 buckets in 11 minutes flat. No, it wasn’t a misconfigured state file. It was a regression in Terraform core’s resource graph pruning logic.

🔴 Live Ecosystem Stats

⭐ hashicorp/terraform — 48,319 stars, 10,333 forks

Data pulled live from GitHub and npm.

📡 Hacker News Top Stories Right Now

How OpenAI delivers low-latency voice AI at scale (118 points)

I am worried about Bun (307 points)

Securing a DoD contractor: Finding a multi-tenant authorization vulnerability (135 points)

Talking to strangers at the gym (954 points)

Formatting a 25M-line codebase overnight (51 points)

Key Insights

Terraform 1.10.0’s resource graph pruning incorrectly marks 1 in 8 dependent resources for deletion when using nested modules with dynamic for_each loops.

The…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.