An example scenario, stated as a scenario
Example scenario (illustrative; not a measured case, and not drawn from a named project): a maintainer of a small library has five pull requests queued and access to a free model endpoint that will review diffs. On the first pull request the endpoint returns six comments; three read plausibly, one references a function that does not exist in the patched tree, and none explains what a complete review would have covered. Nothing in that run defines ground truth, so the maintainer cannot say whether the reviewer is useful or merely fluent.
Public bug-fix datasets solve this problem for their own repositories, not for yours, because they differ in language, defect mix and review conventions. A better-labelled corpus is usually sitting in the repository's own history, since every merged fix is a commit where the defect and its repair are both recorded.β¦