Menu

1 / 8

Amit Got Stuck Reviewing Agent Code: Green CI, Wrong Behaviour

DEV Community·Amit Upadhyay·2 days ago
#UeljOImL
#dev#stroke#fill#color#tests
Reading 0:00
15s threshold

Every check passed. Twelve new tests, green CI, two approvals. And the invoices were rounding down.

A story: the names and numbers are invented, the failure is real.

The pull request looked great. Tidy functions, good names, twelve new tests, all passing. An agent had written it. I read it, liked it, and approved it.

Two weeks later a customer told us their invoices were rounding down. I dug in and found the bug in the new code. Then I opened the new test for it, and the test asserted the wrong value.

I rebuilt what happened:

The agent wrote the code, and the code rounded down.

The agent then wrote a test, and the test asserted what the code did.

The two agreed with each other, so CI went green.

I read a tidy diff and saw nothing that looked like a mistake. Because it did not look like one.

Green CI. Passing tests. A clean diff. And the wrong behaviour shipped anyway. I stared at the test for a long time.

When I'm stuck, I open the one reference that does not hand out answers.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.