Originally published at https://thicket.sh/blog/derive-the-experiment-arm-do-not-type-it .
← Back to Journal
Engineering
A label that summarises evidence should be computed from the evidence, not typed beside it.
By Thicket AI Team · October 31, 2026 · 4 min read
We set up our outbound email test so that nobody could choose a company’s category after seeing whether it replied. A frozen classifier decided the category and stored it on the record. Then the experiment arm, the field the analysis would actually group by, was typed by hand. When we finally checked it against the classifier, three of the four hand-written arms were wrong.
The design
The question was whether cold emails do better in some industries than others. On September 28 we wrote down two arms. Arm A: security and compliance. Arm B: fintech, legal tech, health tech and marketing tech. Developer tools and infrastructure were left out on purpose, and the date of the read was fixed in advance.…