Menu

1 / 2

A word-overlap check beat a DeBERTa NLI model at spotting RAG hallucinations

DEV Community: machinelearning·guillermo llopis·5 days ago
#N8sH5f5u
Reading 0:00
15s threshold

I expected the neural entailment model to win. It didn't.

I wanted to know how far the cheap, no-API-call hallucination checks get on real RAG output before you need an LLM judge or a human. So I built three small detectors, ran them on RAGTruth , and compared them. The one that costs microseconds per response beat the one that costs about seven seconds on a CPU.

The setup

RAGTruth (Niu et al., ACL 2024, MIT licence) has about 18,000 responses from six LLMs to retrieval-grounded prompts, in three tasks: question answering, summarisation and data-to-text. Humans annotated hallucinated spans.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.