Menu

We Typed Our Experiment Arms by Hand. 3 of 4 Were Wrong.
📰

We Typed Our Experiment Arms by Hand. 3 of 4 Were Wrong.

DEV Community: datascience·Yonatan Naor·10 days ago
#R7OIxHpa
#devtodatascience#science#devto#dev#field
Reading 0:00
15s threshold

Originally published at https://thicket.sh/blog/derive-the-experiment-arm-do-not-type-it .

← Back to Journal

Engineering

A label that summarises evidence should be computed from the evidence, not typed beside it.

By Thicket AI Team · October 31, 2026 · 4 min read

We set up our outbound email test so that nobody could choose a company’s category after seeing whether it replied. A frozen classifier decided the category and stored it on the record. Then the experiment arm, the field the analysis would actually group by, was typed by hand. When we finally checked it against the classifier, three of the four hand-written arms were wrong.

The design

The question was whether cold emails do better in some industries than others. On September 28 we wrote down two arms. Arm A: security and compliance. Arm B: fintech, legal tech, health tech and marketing tech. Developer tools and infrastructure were left out on purpose, and the date of the read was fixed in advance.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.