Menu

The timeout that ate the database pool: how Inngest lost every connection twice in one afternoon
📰

The timeout that ate the database pool: how Inngest lost every connection twice in one afternoon

DEV Community: go·Kashif Manzer·13 days ago
#mZePcazS
#devtogo#devto#dev#database#pool
Reading 0:00
15s threshold

At 14:00 UTC on October 10, 2026, both of Inngest's database connection poolers lost every connection to the primary database. Not most of them. All of them. And then, at 15:30, it happened again.

Inngest runs background jobs for other companies. When you trigger a function there, a scheduler has to look up your account and your function before any work can start. Those lookups go through the database. With no usable connections, run scheduling stalled and queue processing stopped for most accounts.

The team restarted the poolers. That did not reliably fix it. They raised the pool size. That did not help either. The pool was not too small. It was being emptied faster than it could be refilled, by the very timeouts meant to protect it.

This is a story about a timeout doing exactly what it was told to do, in exactly the wrong place.

The pooler in the middle

First, the pieces, plainly.

A database connection is expensive to open. So most services do not open a fresh one for every request.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.