Menu

Sampling Ad Exchange Events at Scale: What We Kept, What We Dropped, and Why
📰

Sampling Ad Exchange Events at Scale: What We Kept, What We Dropped, and Why

#hackernoon#technology#volume#spark#sampling
Reading 0:00
15s threshold

At Unity, our ad exchange logged every stage of an auction into Kafka. This included the bid request, bid responses, filters applied, auction decision, fill, impression, and click events. When I looked at the data volume, I found that we were processing around 10 GB per second.

The problem was that every downstream Spark job had to process this entire volume, even when it only needed a small part of the data.

I was the only data engineer in a virtual pod working on this project, along with two product managers, two senior software engineers, and an architect. I proposed sampling the stream and also suggested changes to the Spark clusters running the reports. Here is how we approached the problem and what we learned along the way.

Where was the money actually going?

The data was costing us money at multiple stages.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.