Originally published on Litmiq .
Summaries fail quietly. A model can produce three clean bullet points that read well and still turn "might be covered" into "is covered", round $1,240 up to "about $1,500", or drop the "not" from "decided not to extend hours". Nobody notices until someone acts on the summary.
The research name for this is faithfulness : whether everything in the summary is supported by the source. A large human evaluation of abstractive summarization systems found they were highly prone to generating content unfaithful to the input document ( Maynez et al., 2020 ). Those were earlier models, but the failure types it describes are the ones to test for today. This guide turns them into eight test cases you can run in Litmiq , plus a checklist for reading the results.
Everything below is an illustrative example. The "Orchard Lane Library" committee, its people, amounts and dates are fictional.…