Menu

The database log that filled up and stopped every CircleCI pipeline
📰

The database log that filled up and stopped every CircleCI pipeline

DEV Community: cicd·Kashif Manzer·15 days ago
#6WYmcbcR
#devtocicd#devto#dev#database#disk
Reading 0:00
15s threshold

At 15:48 UTC on October 6, 2026, CircleCI stopped starting work. Not some work. All of it.

You could still trigger a pipeline. The button worked. Nothing happened after it. For 50 minutes, no workflows or jobs started at all. Pipelines triggered from 16:02 failed outright. About 3,500 jobs failed to start during the incident, according to CircleCI's own post incident report. Job start times did not return to normal until 20:30 UTC, five hours after the first delays.

Nothing had been deleted. No code had been deployed badly. The cause was quieter than that, and it had been building for hours before anyone treated it as an emergency. A routine cleanup job filled a disk, and that disk was the one thing every pipeline depended on.

This is a story about a log file. If you run a database, or you depend on one, it is worth five minutes of your attention.

The machine in the middle

When you trigger a pipeline on CircleCI, your request goes to a workflow orchestration service. Think of it as a dispatcher.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.