Menu

Observability That Actually Helps at 3am
📰

Observability That Actually Helps at 3am

DEV Community: go·Vandit Singh·13 days ago
#quIwyZCX
#devtogo#devto#dev#trace#request
Reading 0:00
15s threshold

Here's the moment observability is actually for. It's 3am, one request out of thousands was slow or errored, and it touched eight services on its way through. The user is angry, the dashboard says "p99 latency up," and you have no idea which of the eight did the damage. If your answer to that question is "SSH into each service and grep the logs by timestamp and hope," you don't have observability. You have logs, which is not the same thing.

I spent a stretch wiring real observability across a set of microservices: OpenTelemetry for instrumentation, feeding metrics into Prometheus and traces and logs into Grafana. The tooling is well-trodden and I'll cover it. But the thing that actually changed how fast we could debug wasn't a tool, it was one correlation trick that ties the three signals together.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.