Menu

1 / 3

I spent a week with Claude Code trying to squeeze more out of my memory tool's LongMemEval score. Almost nothing worked. Is it my code or the benchmark?

DEV Community: rust·iCe Gaming·about 13 hours ago
#kJHM3O8T
#devtorust#devto#dev#questions#real
Long article (508 words) — no AI summary yet.
Reading 0:00
15s threshold

I built a local-first memory for coding agents. It's a Rust binary and one SQLite file, using MiniLM embeddings plus BM25 keyword search, with no LLM in the retrieval path and no re-ranker.

On LongMemEval S (470 scored questions) it scores R@1 0.913 (429), R@5 0.987 (464) and R@10 0.996 (468).

I wanted more, mainly at R@1 and recall_all (every gold session in the top K). But all while preserving the strict rules of NO LLM, no re-ranker model and not tuning for the benchmark.

Claude Code did most of the legwork:

offline replays of chunk-score dumps

a DEV/TEST split by question-id hash

a "guard" suite (my coding eval plus probes against a snapshot of my real palace), so a benchmark gain couldn't quietly hurt real use

What we tried

Idea LongMemEval Real-palace guards

Fix two WordPiece tokenizer deviations 0 questions changed probes -1/200

Embed at MiniLM's trained 128 tokens, not 512 R@1 +1, recall_all@5 -7 probes 0.900 -> 0.875, fail

256 tokens +4/-4 at R@1 fail

Re-weight cosine vs BM25 (0.70 to…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.