Garuda is an open-source coding agent for the terminal that I build to learn how coding agents work. Every command runs in an OS sandbox (Seatbelt on macOS, bubblewrap on Linux), edits ask first, and every decision goes to an audit log. It works with Claude, OpenAI-compatible APIs and local models through Ollama, and in editors over ACP.
Garuda had an eval command from its first release, but it gave one run per task and one number. That is enough to catch a regression. It is not enough to say "Garuda is cheaper than X" or "this change helps". For 0.18 I rebuilt the evals so that a claim comes with an interval, and then I tested Garuda against Claude Code, a code-graph MCP server and a tuned prompt.
The method
Repeated runs. Each task runs 3 or 5 times, in a shuffled order from a seed, so a slow period of the API does not hit one task only.
A held-out set with traps. Ten tasks in TypeScript, Python and Java that no tuning ever sees.…