I have been recording how GNU coreutils 9.12 behaves, then replaying the same cases against uutils , the Rust rewrite. The recorder is behave : a Go CLI (and MCP server) that probes flags from --help , odd stdin, missing files, and other corners, then diffs stdout, stderr and exit codes.
So far that work has led to about 30 bug reports and 37 fix PRs upstream. 27 of those PRs are merged. Here is what those merges taught me about black-box testing a rewrite, with concrete examples.
1. Match the reference release, not "whatever is on PATH"
CI on Ubuntu often ships an older GNU than the release the rewrite targets. I record from GNU 9.12 because that is what uutils aims at. A case that is correct on 9.12 can fail on 9.4 (for example how wc or expand counts a multi-byte character). Specs that depend on a version gap stay out of CI, or they teach people to ignore failures.
Lesson: pin the reference binary. Diffing against a random system coreutils produces noise that looks like product bugs.
2.…