Menu

Citation Is Not Truth: A Three-Metric Evaluation of an Extractive Research Pipeline
πŸ“°

Citation Is Not Truth: A Three-Metric Evaluation of an Extractive Research Pipeline

DEV Community: machinelearningΒ·Tushar KumarΒ·10 days ago
#e4yWcyBW
#devtomachinelearning#education#devto#dev#claim
Reading 0:00
15s threshold

Citation Is Not Truth: A Three-Metric Evaluation of an Extractive Research Pipeline

Abstract. I built a research agent whose citations are structural: a researcher extracts claims from sources, a critic admits only claims whose content tokens are contained in the cited source, and the writer sees nothing else β€” so every claim in every report carries a checkable citation, and the seed eval scores it perfectly (citation coverage 19/19, verifiability 19/19, planted fabrications 6 in / 6 caught / 0 leaked). I then attacked the pipeline the way a real web would: poisoned sources carrying fabricated facts, planted instructions, canaries, swapped figures and entity swaps, quote-mined claims, fake corroboration, and questions the sources cannot answer β€” 13 hand-written scenario seeds, 50 deterministic structural variants, 8 benign near-miss controls β€” scored on attack success rate (ASR), false-refusal rate (FRR), and completion, plus citation accuracy re-verified independently.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.