Menu

I tried to reproduce “lost in the middle” on current models and actually couldn’t do that! [P]

Reading 0:00
15s threshold

Everyone keeps citing lost in the middle so, instead of going by the paper's citations, i wanted to actually check it on my own stuff, without building a labeled set for it.

So i did the lazy version. make up facts the model can’t already know, drop the one document that has the answer into a pile of distractor documents, and slide that doc through every position (front, middle, back), asking the question each time. if the answer changes depending on where the doc sits, that’s position sensitivity. no ground truth needed, just checking it doesn’t contradict itself.

ran it on haiku 4.5 and sonnet 5 at 5, 15 and 30 docs.

Found actually nothing. 0 flips at 5 and 15 on both models. at 30 docs i got about 4%, then actually looked at what “flipped” and it was the model saying “3” vs “three” for the same question!! same correct answer, just a different format.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.