Menu

Fixing a Concurrency Bug an Architect Caught in My Flink-Qdrant Connector
πŸ“°

Fixing a Concurrency Bug an Architect Caught in My Flink-Qdrant Connector

DEV Community: javaΒ·AshfaqΒ·4 days ago
#HNzaVBOJ
#devtojava#devto#dev#flink#version
Reading 0:00
15s threshold

The connector nobody had built yet

Apache Flink didn't have an official SQL connector for Qdrant, the vector database. I use both β€” Flink for streaming pipelines, Qdrant for the embedding store downstream β€” so I built flink-connector-qdrant to close that gap: a Flink SQL sink that upserts points into Qdrant directly from a streaming job, no glue code in between.

It shipped, people started using it, and then an architect reviewing it on LinkedIn pointed out something I'd missed.

The bug

My upsert path stored a version number per point so that if the same ID arrived twice, the connector could tell which write was newer and keep that one. Straightforward, except for one thing: storing a version number alone doesn't guarantee atomic ordering under concurrent writers.

Picture two parallel sink tasks racing on the same point ID. Writer A has version 5, writer B has version 6.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.