Menu

The Resilience Feature That Wasn't There
πŸ“°

The Resilience Feature That Wasn't There

DEV Community: microservicesΒ·Ramesh YaraΒ·about 6 hours ago
#IVAVe9l5
#devtomicroservices#devto#dev#resilience#article
Reading 0:00
15s threshold

When I finished Phase 6 of my event-driven microservices platform, I updated the architecture decision record.

ADR-016 described a resilience layer: circuit breakers on outbound dependencies, rate limiting at the perimeter, and β€” the part that felt particularly clean β€” a self-healing DLT pipeline . Dead-lettered events wouldn't sit in a table waiting for manual triage. A scheduled reprocessor would replay them with exponential backoff. A poison-message cap would prevent one bad event from blocking the rest. Dead letters would become an automatically draining queue.

The ADR status: Accepted .

The README described it. The runbook referenced it. I committed both and moved on.

Three months later, I was writing Phase 7 SLOs and noticed a Prometheus gauge I'd added: dlt_unreprocessed . It was reading 32 .

I went to check what the scheduled reprocessor would do with those 32 events. There was no scheduled reprocessor. The table had been growing since Phase 5 failure tests. Nothing had ever drained it.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.