Menu

Fixture Leases: Catch Agent Patches That Pass Because Another Test Ran First
📰

Fixture Leases: Catch Agent Patches That Pass Because Another Test Ran First

DEV Community: cicd·Finley Zhou·8 days ago
#kwZNIYQQ
#devtocicd#devto#dev#test#order
Reading 0:00
15s threshold

Most agent-patch gates measure the wrong thing. They rerun the suite and ask "is it green?" — but a green run only tells you the patch didn't break the order in which the suite happened to execute. Shuffle that order and a meaningful fraction of agent patches change verdict.

This post is about the specific failure I keep hitting: a patch that mutates a shared fixture, then passes because some earlier test already put the fixture in the right state. I'll show a self-contained pytest harness that makes fixture ownership explicit (a lease ), shuffles execution order across seeds, and sorts every test into one of four buckets you can act on.

The failure mode, precisely

Order-dependent passes are cheap for a model to produce. Give an agent a failing test, and the smallest edit that turns it green may write to a fixture that another test wrote first.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.