At Unity, our ad exchange logged every stage of an auction into Kafka. This included the bid request, bid responses, filters applied, auction decision, fill, impression, and click events. When I looked at the data volume, I found that we were processing around 10 GB per second.
The problem was that every downstream Spark job had to process this entire volume, even when it only needed a small part of the data.
I was the only data engineer in a virtual pod working on this project, along with two product managers, two senior software engineers, and an architect. I proposed sampling the stream and also suggested changes to the Spark clusters running the reports. Here is how we approached the problem and what we learned along the way.
Where was the money actually going?
The data was costing us money at multiple stages.…