RemembrallMCP

RemembrallMCP Benchmark Roadmap

Last updated: April 2026

Goal

Benchmark the product surface RemembrallMCP actually exposes:

1. Persistent memory retrieval quality

2. Long-horizon conversational memory behavior

3. Code graph and impact-analysis correctness

4. End-to-end agent productivity on real coding tasks

No single public benchmark covers all four. The benchmark strategy should therefore be a layered evaluation stack, not a search for one headline number.

Product Surface to Benchmark

Current Coverage

RemembrallMCP already has the beginnings of the right benchmark stack.

1. Agent productivity A/B benchmark

• Location: benchmarks/

• Measures: tool calls, estimated tokens, wall clock time, task accuracy

• Strength: directly supports the "agents explore less" product claim

• Gap: current task set is small and handcrafted

2. Parser and impact correctness harness

• Location: crates/remembrall-test-harness/

• Measures: symbol recall/precision, relationship recall/precision, import resolution, impact accuracy, edge cases

• Strength: directly validates the code graph

• Gap: focuses on parser correctness more than full MCP tool behavior

3. Memory recall quality harness

• Location: crates/remembrall-recall-test/

• Measures: Recall@5, Precision@5, MRR, latency, pass/fail thresholds

• Strength: directly validates retrieval quality

• Gap: limited coverage for temporal updates, stale memories, abstention, and multi-session synthesis

External Benchmarks: Fit Assessment

The right question is not "what is the industry standard benchmark for memory?" but "which public suites add signal on top of our product-specific harnesses?"

Recommendation

Use a four-layer benchmark stack:

1. Keep custom evals as primary

   • They are the only benchmarks aligned to the full Remembrall product surface.

2. Add SWE-bench Verified for external coding-agent validation

   • This is the strongest independent check on the productivity claim.

3. Add one long-memory benchmark

   • Prefer LongMemEval.

   • Use LoCoMo only if we want a second conversational-memory view later.

4. Optionally add a generic retrieval regression suite

   • Prefer a narrow BEIR slice instead of a large generic benchmark rollout.

What Each Benchmark Should Prove

A. Code graph correctness

Question: "Does the graph return the right symbols and dependency edges?"

Owner: crates/remembrall-test-harness

Success metrics:

• Symbol recall >= 95% on small fixtures

• Relationship recall >= 85% on small fixtures

• Import resolution >= 90%

• Impact query recall >= 95% for direct callers

• No language below grade B on supported languages

Release gate:

• Any drop >5 points on a language dimension triggers investigation

• Any impact-analysis regression on must_find relationships blocks merge

A2. Field-level reference correctness

Question: "Does the graph capture data fields and resolve who reads them - the 'whole codebase' claim?"

Owner: crates/remembrall-test-harness with test-fixtures/field-capture.toml

Why it matters: Field-level capture is what makes the whole-codebase positioning truthful. ORM-heavy codebases live and die by "who references the amount field?" - a question symbol-only graphs cannot answer.

Success metrics:

• Field symbol recall: every class/struct data field in the fixture is captured as a Field symbol with the right parent_symbol_id

• Defines edge recall: class/struct -> field edges are all found

• References edge recall/precision: self/receiver field reads (self.x, this.x, &self.x, Go s.X, Ruby @x) resolve to the enclosing type's field

• No-fan-out correctness: ambiguous cross-file field names are dropped, not fanned out one-edge-per-candidate

• Upstream impact on a field returns the methods that read it

Release gate:

• Any regression on field References recall blocks merge

• A new supported language must ship a field-capture fixture case before it is claimed as supported in the README

B. Memory retrieval quality

Question: "Does remembrall_recall return the right memories quickly?"

Owner: crates/remembrall-recall-test

Success metrics:

• Aggregate Recall@5

• Aggregate Precision@5

• MRR

• P50/P95 latency

• False positive rate for "should abstain" queries

• Filter correctness for memory_type, tags, and project

Release gate:

• No regression in pass rate

• No increase in false positives for abstention queries

• P95 latency stays within a defined tolerance band

C. Long-horizon memory behavior

Question: "Does memory remain correct across sessions, updates, conflicting facts, and time?"

Owner: new harness under benchmarks/long_memory/ or a dedicated crate if execution logic grows

Preferred benchmark: LongMemEval

What to measure:

• Information extraction

• Multi-session reasoning

• Temporal reasoning

• Knowledge updates

• Abstention

How to adapt it for Remembrall:

• Run the benchmark through a thin adapter that maps session facts into remembrall_store

• For update scenarios, explicitly use remembrall_update or store superseding memories and verify stale facts stop winning

• Score both retrieval quality and final answer quality

Pass criteria for adoption:

• Benchmark can run reproducibly from the repo

• Results are stable enough across repeated runs to detect regressions

• At least one metric maps directly to Remembrall behavior rather than only model behavior

D. End-to-end coding-agent productivity

Question: "Do agents finish real engineering tasks faster, cheaper, or more accurately with Remembrall?"

Owner: benchmarks/

Preferred benchmark: SWE-bench Verified plus current handcrafted A/B tasks

What to measure:

• Task success rate

• Cost proxy: tokens and tool calls

• Latency / wall clock

• Retrieval usage patterns

• Failure modes: missed callers, wasted exploration, bad stale-memory retrieval

Why both are needed:

• Current handcrafted tasks isolate the graph value proposition clearly

• SWE-bench validates that those gains survive in messy, real-world task flows

Phased Rollout

Phase 0: Stabilize current benchmark story

Status: in progress

Work:

• Keep benchmarks/ as the primary marketing benchmark

• Keep parser and recall harnesses as release-safety benchmarks

• Add this roadmap and align docs around the layered strategy

Exit criteria:

• Benchmark surfaces are documented

• Each existing harness has explicit ownership and intended use

Phase 1: Expand memory-recall harness for time and updates

Why first: This is the cheapest high-signal improvement because the harness already exists.

Work:

• Add recall cases for superseded facts

• Add cases for contradictory memories with timestamps

• Add abstention cases where semantically related memories should not be returned

• Add cases that require combining two related memories from different sessions

• Add report fields for false-positive rate and stale-fact rate

Suggested new categories:

• G: temporal updates

• H: contradictions and supersession

• I: abstention

• J: multi-memory synthesis

Deliverables:

• expanded tests/recall/ground_truth.toml

• scorer updates for stale-fact and abstention failures

• seed fixture additions

Phase 2: Add LongMemEval-inspired adapter

Why second: Only after the local harness tests the same failure modes.

Work:

• Vendor or document a pinned LongMemEval version

• Build a runner that converts benchmark sessions into memory writes

• Evaluate two modes:

  • retrieval-only: inspect whether the right memory items are surfaced

  • answering: use a fixed model and prompt to answer benchmark questions from retrieved memories

• Publish a concise result table in benchmarks/reports/

Important constraint:

Do not present this as "the" Remembrall benchmark. It covers the long-memory slice, not the code graph.

Phase 3: Add SWE-bench Verified A/B

Why third: Highest upside, but operationally heavier.

Work:

• Pick a small pinned subset of SWE-bench Verified tasks first

• Define a reproducible with/without-Remembrall protocol

• Require fresh conversations and pre-indexed repos for the with arm

• Record:

  • solve rate

  • time to first correct patch

  • total tool calls

  • total tokens

  • whether Remembrall tools were actually used

Start small:

• 10 to 25 tasks

• 2 or 3 representative repos

• one supported language first, then expand

Success criteria:

• measurable increase in solve rate or equivalent solve rate with materially lower tool cost

Phase 4: Optional generic retrieval baseline

Why optional: Useful for engineering discipline, weak for product storytelling.

Work:

• Run a narrow BEIR slice against the memory search layer

• Use it as a regression dashboard, not a homepage claim

Reporting Model

Keep benchmark reports split by audience.

Engineering reports

Purpose: catch regressions and guide implementation work

Include:

• per-category scores

• failure cases

• latency percentiles

• benchmark version and dataset pinning

Product and marketing reports

Purpose: support claims without overselling benchmark scope

Include:

• A/B coding-agent savings from benchmarks/

• parser quality summary by language

• memory recall summary

• external benchmark slices with tight scope labels

Avoid:

• collapsing everything into one synthetic "benchmark score"

• calling LongMemEval the product benchmark

• claiming broad superiority from retrieval-only results

Repository Changes to Prioritize

Near term

1. Expand remembrall-recall-test to cover temporal memory behavior

2. Add a machine-readable benchmark manifest with benchmark name, version, scope, and owner

3. Add report templates so results are comparable across runs

Mid term

1. Add benchmarks/long_memory/README.md

2. Add LongMemEval adapter and pinned instructions

3. Add a small SWE-bench Verified subset harness

Nice to have

1. Trend charts for benchmark regressions over time

2. CI mode for lightweight subsets

3. Benchmarks that isolate the value of remembrall_update and stale-memory suppression

Decision Summary

Use this stack:

• Primary product benchmarks: existing custom A/B benchmark, parser harness, recall harness

• Best external benchmark to add: SWE-bench Verified

• Best long-memory benchmark to add: LongMemEval

• Optional retrieval regression benchmark: BEIR slice

That keeps the benchmark program aligned to the actual product instead of drifting toward benchmark theater.