Menu

I benchmarked my own agent skill, and today's Claude barely needed it
📰

I benchmarked my own agent skill, and today's Claude barely needed it

DEV Community: datascience·Furkan Boran·16 days ago
#SHUUnvNp
#devtodatascience#science#devto#dev#skill
Reading 0:00
15s threshold

I wanted to write an agent skill that makes coding agents better at data analysis. The kind of mistakes I had in

mind are the ones that produce a confident, wrong number: a "Total" row summed together with the countries, a join

that duplicates orders, averaged p99s, timed-out requests that vanish from the latency percentiles, an A/B test that

was supposed to split 50/50 and came back 50.7/49.3.

Every rule in the skill comes from a paper or official documentation. But before claiming it helps, I wanted to

measure it.

The benchmark

I built 30 realistic cases, each with one planted trap and a correct answer computed by code:

19 data-analysis traps (subtotal rows, join fan-out, average of averages, NA parsed as null when it's Namibia, partial weeks, sample ratio mismatch, segment fishing, Simpson's paradox, …)

8 performance and reliability traps (closed-loop load tests, a saturated load generator, retries counted as requests, tail-sampled traces, regression to the mean after a "perf sprint", …)

3 tasks…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.