Dimple is a local-first memory system for AI agents. It runs as an SDK, a CLI, or an MCP server. You give it chat turns, notes, and documents, and it returns the items that matter when an agent asks a question.
This post reports one run against LongMemEval_S, a public benchmark of 500 questions about long chat histories. Storage, embeddings, and retrieval ran locally. A separate reader model answered from the retrieved window, and a judge scored each answer. The reader and the judge ran remotely.
Dimple answered 454 of the 500 questions correctly, 90.8%.
The score matters less than its source. This post separates the misses that retrieval caused from the misses the reader caused. A receipt is the record for one question: the reference answer, the reader’s answer, and where the evidence sat.
The judge marked 46 answers incorrect. 42 are answerable: 22 belong to the reader and 20 to retrieval. The other 4 are abstention variants, described below.
How the run works
Each bar is one question, grouped by type in six lanes. Hover a bar for its receipt, or tap it on a phone. The arrows step through a lane, and replay runs the whole run.
LongMemEval
454 / 500
questions answered correctly · 90.8%
one bar = one question
- judged correct
- judged wrong
What to notice. The failures are not equal between the lanes: multi-session holds 24 of the 46 judged wrong and its score is the lowest at 82.0%. Its questions also need the most evidence conversations per answer, 2.61. This is the pattern, with one exception: preference needs one conversation and its score is still 86.7%.
Each question ran in its own store, and ingest never saw the question. One turn becomes one memory, with the conversation id, date, turn index, and role. The embedding is Xenova/all-MiniLM-L6-v2 at 384 dimensions.
A query fuses vector and keyword results with reciprocal rank fusion, a standard way to merge two ranked lists. Dimple takes the top 50 turns and the first 10 whole conversations in that ranking. The reader sees those 10 only.
The reader and judge are both deepseek-v4.1-flash at temperature 0, using the benchmark’s prompts. One run, on SDK 0.0.0-alpha.11. The harness, the report, and the artifacts are public.
Where Dimple is strong
The two single-session lanes have 2 misses together. The assistant lane answered 56 of 56 questions. The user lane answered 68 of 70. Both ask for one fact from one conversation.
The ranking can also be scored alone. The table gives two measures at several depths. A needed item is an evidence turn or conversation that the answer requires.
| what is ranked | k | at least one found | every one found |
|---|---|---|---|
| whole conversations | 5 | 97.0% | 86.8% |
| whole conversations | 10 | 98.9% | 93.8% |
| individual turns | 5 | 75.3% | 42.6% |
| individual turns | 10 | 85.1% | 60.6% |
| individual turns | 50 | 96.6% | 89.6% |
In the first row, 97.0% have at least one needed conversation in the top five conversations. 86.8% have all of them. The reader’s window is larger, at k=10 conversations, so both are higher.
The temporal lane (124 of 133) and the knowledge lane (71 of 78) come next. With no answer in the history, the reader scores 26 of 30 (86.7%). With one, it scores 428 of 470 (91.1%). The first group has only 30 questions, so the difference supports no conclusion.
Where it is weaker
Multi-session is the weakest lane. These questions need more than one conversation, and the answer is usually a single number. The lane answered 109 of 133 (82.0%) and owns 24 of the 46 judged misses.
The preference lane is lower: 26 of 30 (86.7%). It asks the same question type as the single-session lanes, and it is the smallest lane.
The 42 answerable misses split between reader and retrieval: 22 and 20. A miss belongs to the reader when the evidence was already in its window. It belongs to retrieval when the window lacked evidence.
The attribution names the half that made a correct answer impossible. A reader can still make a second error inside the window.
| question type | answerable misses | misses the reader caused | misses retrieval caused |
|---|---|---|---|
| assistant | 0 | 0 | 0 |
| user | 2 | 1 | 1 |
| temporal | 9 | 1 | 8 |
| knowledge | 6 | 6 | 0 |
| preference | 4 | 3 | 1 |
| multi-session | 21 | 11 | 10 |
| all six | 42 | 22 | 20 |
The table counts the 42 answerable misses. The other 4 are abstention variants. Each copies a question but removes its answer, so the correct answer says the information is absent. 42 plus 4 gives 46.
The multi-session row works the same way: 21 plus 3 variants give 24. The fourth variant is a knowledge question, which is why that lane shows 7 judged misses and the table shows 6.
Two receipts are clear failures: a bus fare the history cannot give, and an engineer count taken from the wrong moment. The other two say what their references say in different words.
One asks how many times the user baked egg tarts. The reader answered “0 times. I don’t see any mention of you baking egg tarts in the past two weeks”, and the judge did not accept it.
Two of the 46 misses depend on wording. The four ids end in _abs: 88432d0a_abs, 80ec1f4f_abs, 09ba9854_abs, and 031748ae_abs.
Why the misses happened
Of the 42 answerable misses, 22 had all evidence in the window. The other 20 belong to retrieval, from three causes:
- 9 had all evidence in the pool, but the 10-conversation window dropped some.
- 9 had only part of the evidence in the pool.
- 2 never retrieved the needed conversation.
The answers were already in the pool of 50 turns, and the ranking found them.
The lanes fail for different reasons:
- Knowledge updates. 6 misses, all the reader’s. Both versions of the changed fact were in the window, and the reader used the earlier one.
- Temporal reasoning. 9 misses, 8 retrieval’s. One absent conversation makes the answer wrong.
- Preference. 4 misses, 3 the reader’s. The reader gave general advice instead of the preference.
- Multi-session. 21 misses, 11 the reader’s and 10 retrieval’s.
One knowledge question asks for a mortgage pre-approval. The history has $350,000 and a later $400,000. The reader gave the first and mentioned the second. It did not treat the later statement as current.
Two causes of a wrong multi-session answer
Counting. Question 88432d0a asks how many times the user baked something in two weeks. The answer is 4. The reader received 10 conversations that describe those 4 bakes 11 times, and all 4 evidence conversations were among them. It answered 8.
It counted each bake once, counted 3 a second time, and added an apple pie. Retrieval was not at fault.
How many times did I bake something in the past two weeks?
8 / 4
answered / the reference
The same 4 bakes appear 11 times in 10 chats.
one mark = one mention
- counted in the reference answer
- mentioned again
- where the answer comes from
11 mentions of 4 bakes across 10 chats. Press replay, or step one mention at a time.
- bake
- the daymentions
- sourdough bread
- Tue 16 May5
- whole wheat baguette
- Sat 20 May2
- convection cookies
- Thu 25 May2
- chocolate cake
- 27-28 May2
It listed all 4 bakes once, sourdough bread, chocolate cake and whole wheat baguette a second time, plus an apple pie.
Completeness. A multi-session question needs 2.61 evidence conversations on average. Of the 133, retrieval finds at least one needed turn for 73.6% and every needed turn for 18.2%. The turn rows in the recall table cover all 470 questions, so their numbers are higher.
Retrieval ranks turns, not conversations, so one long conversation can use many of the 50 positions. The top 5 turns span 2.52 conversations. The top 50 span 13.54.
Question 6d550036 needs 4 evidence conversations. 3 reached the pool. The reader answered 7 against a reference of 2.
It listed the projects in the window. By the rule above, the miss belongs to retrieval. The overcount is the reader’s own.
What the numbers do not say
- One run, one configuration. Temperature 0, one reader, one judge, no seed or judge-variance study. The reader and judge are the same model. A difference of one or two points between lanes means nothing alone.
- No full-context control. Give the same reader the whole history and the same judge. Until then, this measures a system, not the value of retrieval in it.
- Not fully local. The reader and judge ran remotely.
- Not comparable to published tables. Other readers and judges produced those.
- The reading method. The published baselines used chain-of-thought. This run used
direct, with the benchmark’s text byte for byte.
What we are changing next
1. Build the reader’s window from ranked conversations. 9 of the 20 retrieval misses had the evidence in the pool. Rank whole conversations, then use that ranking to fill the window. This changes retrieval, not the reader.
2. Collapse restatements into one event. The reader-owned multi-session misses include answers that counted the same event twice. The next pass treats a second mention as evidence for the existing event, not as a new event. The other 10 reader-owned multi-session misses are the first test.
Provenance
Dimple on npm · Harness · Report · Raw artifacts · LongMemEval · Dataset revision · Official scriptsLongMemEval is the benchmark, and its authors built it. This is one configuration of Dimple against it. Every number is read from the repository at data version 1.1.0, and each value records its artifact.