Menu

We Benchmarked Our Extraction Pipeline Four Ways. The Obvious Knobs Lied.
📰

We Benchmarked Our Extraction Pipeline Four Ways. The Obvious Knobs Lied.

DEV Community: opensource·Paragon666·2 days ago
#vc7yNtjm
#devtoopensource#devto#dev#quota#chunk
Reading 0:00
15s threshold

We are a small independent lab in Los Angeles, and this is the lab notebook

of one real production job — not a synthetic benchmark. The job:

verbatim-quote extraction over a fixed private text corpus. For each chunk,

the model returns up to a fixed quota of exact quotes with anchors. We ran

that job several ways on public models and public software — a local 9B

model on a consumer GPU, and two hosted flagships — and measured the things

that decide whether the output is usable: ground-truth completeness (share

of the reference quote set recovered), parse failure rate, the non-verbatim

rejection share (rejected quotes over all returned quotes), and time.

Within each test the arms share inputs and scoring; where a test uses a

different metric, it says so on the spot. The headline claims travelling

with this stack did not survive contact with a real task.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.