Menu

Withdrawing, and Sharing Everything We Measured: 1,800+ Trials of a 4B Quantized SWE Agent
📰

Withdrawing, and Sharing Everything We Measured: 1,800+ Trials of a 4B Quantized SWE Agent

DEV Community: machinelearning·Japan Open Data Lab·1 day ago
#yGRFEt8B
Reading 0:00
15s threshold

TL;DR — After 1,800+ evaluation trials across 9 system generations, we withdrew from the Gemma 4 competition and are publishing our measurements instead.

Our best custom system scored 15/114 (13.2%) — one task behind the organizer's simplest zero-think baseline ( 16/114, 14.0% ), and never ahead of it.

Longer thinking actively hurt the 4B agent: the think=4096 baseline scored 10/114 vs 16/114 for zero-think (−37.5% relative).

On the unseen holdout, our custom systems (5–6/44) consistently beat the zero-think baseline (4/44) — the machinery helped there, just not enough overall.

Hi everyone — we're Japan Open Data Lab. After 1,800+ evaluation trials across 9 system generations, we've decided to withdraw from this competition, and we'd rather publish our measurements than let them sit in a drawer. Everything below is aggregate results and findings only — no task-level data, no patches, no submission code.

Setup.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.