2 a.m.

An API token expired overnight. That was the whole root cause: one credential, for one

third-party API, reached the end of its life while nobody was looking.

The consumer that depended on it was not careless code. It was built the way the tutorials

and the postmortems say to build one. It caught exceptions and logged them with a stack trace.

It retried, with a limit, so no message could go around forever. Past the limit, the broker

dead-lettered the message instead of dropping it. By every checklist I knew, it was a

well-built consumer.

It did exactly what it was built to do. Every message called the API, was refused, logged an

error, went back on the queue, and tried again. Each message found out about the expired token

on its own. Each one used up its retries on its own. Then each one went to the dead-letter

queue, on its own.

By morning the dead-letter queue held 40,000 messages, and every one of them was customer data

that should have reached the vendor and hadn't.…