When you build a language with two ways to run a program, comparing them catches disagreement but not shared mistakes. For Chuks v0.2.0-rc.3, we added tests that hold programs to a known answer, generated about 7,800 of them from every combination of three language features, and committed to shipping with no further release candidate.

The problem with checking a thing against its twin

Chuks runs programs two ways. A bytecode VM gives instant startup while you develop. A native compiler produces the binary you ship. The promise is simple: the same program prints the same thing in both.

The natural test is to run everything both ways and compare. It is also a trap. Imagine the two implementations share a front end, the part that reads your code and decides what it means. If that part makes a mistake, both back ends faithfully execute the mistake, agree with each other, and the test passes.

We hit this for real. Both back ends once agreed on a value that should have been a typed record and was not.…