Menu

1 / 4

Your RAG doesn't need a better embedding model. It needs better chunking.

DEV Community: rust·iCe Gaming·17 days ago
#M4vD6QFS
#devtorust#devto#dev#session#question
Reading 0:00
15s threshold

For the last week I've been running LongMemEval against Palace, a local memory store I build for coding agents. It's a single Rust binary with SQLite, BM25 and a small ONNX embedding model. No LLM sits anywhere in the write or recall path.

This post isn't about the final number. It's about what I learned getting there: which changes moved recall, which didn't, and how many of the improvements turned out to be plain bugs that only a benchmark run would surface.

Short version, so you can stop here if you want:

Swapping embedding models did nothing on this dataset. Every difference was 1 to 4 questions.

Chunking conversations the way the product actually stores them helped more than anything else.

The ranking gains were mostly bug fixes. A substring match, an overeager phrase rule, and a chunker that silently dropped text.

int8 models give different vectors depending on what else is in the batch. That one surprised me.

What's being measured

LongMemEval asks questions about a long chat history.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.