One lock we didn't even know was contended cost us 8.6 milliseconds — on a track whose entire run, by the end, was 27. That's close to a third of our whole budget, hiding behind a registry that looked completely fine on every machine we could afford to test on, because at two threads there was nothing there to find. We only caught it the way we caught almost everything that actually mattered that weekend: by shipping the wrong answer on purpose, dozens of times, failing our own hackathon's one unforgivable rule just to get the one machine that actually judged us to tell us the truth.
Speed Demon, run by Hackathon Raptors, builds your Docker image and runs it five times on one dedicated Intel box, drops the highest and lowest, and the mean is the only number that counts. For most of three days, that was the only window into it any of us had.…