Menu

It Passed CI. That Still Doesn't Make the Patch Safe.
πŸ“°

It Passed CI. That Still Doesn't Make the Patch Safe.

DEV CommunityΒ·Jordan HuangΒ·about 9 hours ago
#l4nXVJV0
#devcommunity#dev#test#model#article
Reading 0:00
15s threshold

Last week I merged a model-written patch in four minutes.

CI was green. Review said 'lgtm'. The diff was twelve lines.

Two days later a background job stopped retrying. Nobody noticed for a while.

That is the whole problem in one story. A green check feels like a verdict. It is not one.

I generate patches with models every day now. It changed how I merge them.

Below is the FAQ I keep answering, a table you can steal, and a gate script you can run before lunch.

The short version

Claim What the check actually proves Corrected model

'CI is green' the tests you already had still pass green means no regression in covered paths

'The diff is small' few lines moved risk lives in the files touched, not the line count

'I added tests' assertions exist a test written by the same author grades itself

'It looks idiomatic' style matched the codebase idiom hides semantics, and semantics fail

Keep that table open. Now the details.

Claim 1: 'Green CI means the change works.'

No.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.