Menu

I re-ran a rival's memory benchmark. It tied mine. Here is the interesting part.
📰

I re-ran a rival's memory benchmark. It tied mine. Here is the interesting part.

DEV Community: security·Calas Automations·about 5 hours ago
#fREQdA12
#devtosecurity#devto#dev#teams#monomem
Long article (286 words) — no AI summary yet.
Reading 0:00
15s threshold

I build Monomem, shared memory for AI coding agents. It records which app wrote each note and holds suspicious writes until a person looks.

On LongMemEval-S, a standard long-term-memory test, a well-known memory project (agentmemory) publishes 84.7%. I re-ran its own evaluation on the same data file and got 84.7% back, exactly.

Monomem scored 86.0% (404 of 470 questions).

Question by question, 14 were answered only by Monomem and 8 only by agentmemory.

That gap is not significant (exact McNemar test, p = 0.29). It is a tie.

Two more things I will not hide:

Monomem's ranking was tuned on half of this question set, so the fairer figure is the other half: R@5 97.5, Rall@5 86.2. The repo says so.

Another system, Vestige, reports a higher score than mine (its own number, which I have not re-run). It is named in the docs.

Why post a tie? Because memory search is turning into a commodity.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.