One result from the "Lost in the Middle" paper should change how you build RAG prompts. GPT-3.5-Turbo was given 20 retrieved documents with the answer placed in the middle of them, and it scored lower than when it got no documents at all. Answering from memory beat answering with the documents. Here the extra context made the answer worse.

Liu et al. (Stanford, UC Berkeley, Samaya AI, 2023) tested multi-document QA and key-value retrieval. They moved the relevant passage from the first slot to the last and tracked accuracy. The result was a U-shaped curve: strong at both edges, down by 20+ points in the middle. Newer models have flattened that curve, but it hasn't gone away.…