A green test run tells you one thing: the assertion you wrote now holds. It says nothing about the eleven lines the agent changed to get there β and those eleven lines are what your reviewer has to read at 5 p.m. on a Friday.
So I stopped scoring agents on pass/fail. Pass/fail is a coin, not a measurement. What I score now is patch width: how much of the diff lands outside the function that actually broke.
Pin the span before you look at the diff
You can't measure width without an origin. The origin is the innermost function containing the line in the traceback. Don't eyeball it β parse it.
#!/usr/bin/env python3 # scope.py <repo> <path> <line> -> {"path", "start", "end"} import ast , json , sys
repo , path , line = sys . argv [ 1 ], sys . argv [ 2 ], int ( sys . argv [ 3 ]) full = f " { repo } / { path } " tree = ast . parse ( open ( full ). read (), filename = full )
best = None for node in ast . walk ( tree ): if isinstance ( node , ( ast . FunctionDef , ast .β¦