Menu

I tested my AI bug hunter on 44 real bugs from Express, ESLint, Hexo and others. It caught 32.
📰

I tested my AI bug hunter on 44 real bugs from Express, ESLint, Hexo and others. It caught 32.

DEV Community: javascript·billbin1969-debug·8 days ago
#pNhVb9fw
#devtojavascript#devto#dev#code#change
Reading 0:00
15s threshold

AI writes more and more of our code, and code review can't keep up. So I built BuildProof : a tool that looks at a code change and tries to prove a bug, not just guess one.

How it works

The AI reads the change and suspects a bug.

It writes a test, and that test runs on the old code AND the new code.

If the test passes on the old code and fails on the new one, that's a real behaviour change.

A separate AI "judge" decides whether that change was intended or a bug.

You only get bugs that come with a failing test as proof.

The test

BugsJS is a public dataset of real bugs from real projects (Express, ESLint, Hexo, Karma, Shields, Bower, hessian.js). I re-introduced each bug and let the hunter look at the change, with no hints.

Results

Caught 32 of 44 real bugs.

In the reverse direction (the actual fix, which is correct code), it raised a false alarm 4 times out of 44.

Cost: about $0.15 per check.

It also knows when to stay quiet: I ran it on 4 open pull requests in lodash, koa and axios.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.