Tail Sampling with OpenTelemetry, Measured: It Keeps What Matters Until You Scale It or Shrink It
Logs, Metrics and Traces Are Not Three Pillars. They're One Request. ended on an uncomfortable number. A 10% head sampler decides whether to keep a trace when the request arrives , before anyone knows it will be slow. So of 20 slow checkouts, it kept about 2, and the histogram that told you about the slow ones pointed at traces that no longer existed. I promised the fix in that post, and again in the structured logging one : decide at the end of the trace instead. That is tail sampling, and this is the post.
Tail sampling does what it promises. On my run it kept every slow trace and every failed one, while dropping 90% of the rest.…