Menu

Can CI check whether an AI agent actually ran the tests it claims?
📰

Can CI check whether an AI agent actually ran the tests it claims?

DEV Community: cicd·rambo·about 10 hours ago
#9Sijql3h
#devtocicd#devto#dev#receipt#agent
Reading 0:00
15s threshold

Yes, but not by reading the agent's report. CI can check a run the way an auditor checks a set of books: by examining records the run left behind as it executed, records the agent cannot rewrite afterward. There are three ways to do it, weakest to strongest.

1. Artifacts and timestamps (weak, better than nothing)

A real test run touches disk: a test cache, a coverage report, a results file with a fresh timestamp. If the agent claims a run two minutes ago and the newest test artifact is from yesterday, the claim fails. This catches lazy fabrication. It misses an agent working in an ephemeral container, and a stale artifact proves less than a fresh one, so treat this layer as a smell test.

2. A canary test that must fail (strong)

Plant one test that cannot pass, named so a human spots it instantly: a test asserting something false on purpose. A real run of the suite reports that failure. An agent that reports all green without mentioning the canary did not run the suite.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.