I wanted to write an agent skill that makes coding agents better at data analysis. The kind of mistakes I had in
mind are the ones that produce a confident, wrong number: a "Total" row summed together with the countries, a join
that duplicates orders, averaged p99s, timed-out requests that vanish from the latency percentiles, an A/B test that
was supposed to split 50/50 and came back 50.7/49.3.
Every rule in the skill comes from a paper or official documentation. But before claiming it helps, I wanted to
measure it.
The benchmark
I built 30 realistic cases, each with one planted trap and a correct answer computed by code:
19 data-analysis traps (subtotal rows, join fan-out, average of averages, NA parsed as null when it's Namibia, partial weeks, sample ratio mismatch, segment fishing, Simpson's paradox, …)
8 performance and reliability traps (closed-loop load tests, a saturated load generator, retries counted as requests, tail-sampled traces, regression to the mean after a "perf sprint", …)
3 tasks…