Menu

One Hub to Watch Them All: Multi-Tenant Monitoring for 4 Clusters and 400+ Edge Machines with Grafana, Mimir, Loki and Alloy
📰

One Hub to Watch Them All: Multi-Tenant Monitoring for 4 Clusters and 400+ Edge Machines with Grafana, Mimir, Loki and Alloy

DEV Community: kubernetes·Syed Muhammad Jamal Uddin Gilani·4 days ago
#AwRXYYWE
#devtokubernetes#devto#dev#edge#gateway
Reading 0:00
15s threshold

"How I built a self-hosted, GitOps-managed observability hub on Kubernetes: Mimir for metrics, Loki for logs, Alloy as the only agent, tenants enforced at the gateway, and S3 as the real database."

The problem

Our monitoring was spread across too many places. We run a handful of on-prem Kubernetes clusters (one "hub" plus several "spoke" clusters) and more than 400 edge machines at remote sites. Each place had its own partial monitoring. The rest had none.

That caused the usual pain:

No single place to look. When something broke at an edge site, we SSH'd in and grepped logs. Live, if the machine was still reachable.

No history. Local Prometheus instances kept a few days of data on whatever disk they had. Questions like "when did this start?" had no answer.

No separation. If several teams shared one Prometheus, they all saw each other's data, and one noisy sender could hurt everyone.

Hand-made config.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.