Menu

Ops analytics from an event stream: Kafka into ClickHouse without losing or double counting alerts
📰

Ops analytics from an event stream: Kafka into ClickHouse without losing or double counting alerts

DEV Community: go·Awiros·5 days ago
#p1mQaNyK
#devtogo#devto#dev#insert#consumer
Reading 0:00
15s threshold

Everything in this post is made up for illustration: the site, the cameras, the event ids, the timestamps, the counts. The problem is one we deal with building dashboards for Vision AI. On Aura every detection arrives with video evidence, gets routed to the team that acts on it, and rolls up into dashboards across sites. The routing side wants a transactional database: an alert gets raised, acknowledged, attached to a clip, closed. The dashboard side wants "PPE violations per dock per hour for the last 90 days", and asking the operational database for that during a busy shift is how alert pages get slow.

The pattern we'd recommend is a separate analytics path. Apps publish every alert and stat event to Kafka, a small consumer batches them into ClickHouse, and dashboards only ever read ClickHouse. The consumer is maybe 200 lines of Go.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.