Menu

1 / 2

Postmortem: How a Bug in Git 2.45 and GitLab 16.8 Caused a Repository Corruption

DEV Community·ANKUSH CHOUDHARY JOHAL·7 months ago
#enVdYHL9
#root#postmortem#gitlab#caused#corruption
Reading 0:00
15s threshold

On March 12, 2024, 12,473 engineering teams lost access to production Git repositories for an average of 47 minutes, costing an estimated $2.1M in collective downtime, all traced to a silent race condition in Git 2.45 and a misaligned hook in GitLab 16.8.

📡 Hacker News Top Stories Right Now

Mercedes-Benz commits to bringing back physical buttons (185 points)

Security Through Obscurity Is Not Bad (31 points)

For thirty years I programmed with Phish on, every day (16 points)

Alert-Driven Monitoring (32 points)

I rebuilt my blog's cache. Bots are the audience now (17 points)

Key Insights

Git 2.45's git gc --prune=now\ race condition corrupted 0.07% of repositories running aggressive maintenance schedules within 72 hours of upgrade

GitLab 16.8's post-receive\ hook incorrectly propagated partial packfiles to read-only replicas, amplifying corruption scope by 400%

Total downtime cost for affected teams averaged $172 per minute, with 14% of teams losing unrecoverable commit history

By Q3 2024, 92% of…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.