There’s a famous saying in engineering: "Fast, Cheap, or Reliable. Pick two." But when you enter a 48-hour hackathon to build a distributed incident management system, you are forced to pick all three.
This is the story of how my team and I built SmartPager from scratch, moving from a blank IDE to a production-grade alerting system in a single weekend.
The Problem with Incident Management
Incident management isn't just about sending an email when a server goes down. It's about concurrent event handling, real-time escalation, and ensuring that alerts trigger in sub-seconds. If the alerting system itself fails, it's useless. We needed a system that could handle failure scenarios gracefully.
Why Microservices?
The easiest route in a hackathon is a monolith. But we wanted to build something that mirrored real-world production environments.…