We are a small independent lab in Los Angeles, and this is the lab notebook
of one real production job — not a synthetic benchmark. The job:
verbatim-quote extraction over a fixed private text corpus. For each chunk,
the model returns up to a fixed quota of exact quotes with anchors. We ran
that job several ways on public models and public software — a local 9B
model on a consumer GPU, and two hosted flagships — and measured the things
that decide whether the output is usable: ground-truth completeness (share
of the reference quote set recovered), parse failure rate, the non-verbatim
rejection share (rejected quotes over all returned quotes), and time.
Within each test the arms share inputs and scoring; where a test uses a
different metric, it says so on the spot. The headline claims travelling
with this stack did not survive contact with a real task.…