Does AI Memory Actually Work for Coding Agents?
We benchmarked that question three times, and it lost the first round. Here's the full picture.
Memory doesn't make a model smarter — it stops the model from re-learning what it already worked out. That single idea explains every number below, including the ones that didn't go our way.
Markus Sandelin — Independent Research, February 2026
The shape of the answer
Three benchmarks at increasing scale. Memory lost on the smallest one, and won as the codebase and the tasks got bigger.
On a toy codebase, the version without memory won. The codebase was small enough to hold entirely in the model's working context, so there was nothing genuinely worth remembering — every time the agent checked memory, that added cost and latency with nothing to show for it. Most benchmark reports would bury a result like this. We put it first, because a “bad” result that cleanly proves the boundary of our own claim is stronger evidence than a good one — it shows exactly where memory stops being useful, not just that it sometimes is.
On a real production codebase — 4,895 lines, 158 files — the two versions produced work of the same quality; neither one did better. What changed was the effort it took to get there: with memory, the agent took fewer turns, explored the codebase less, and didn't have to re-read files it had already seen in an earlier session.
On a six-agent swarm, the lead agent stopped exploring the codebase altogether. It recalled the architecture and went straight to delegation.
For the weakest model in the test, memory wasn't a matter of efficiency — it was the difference between failing the task completely and nearly acing it. Without memory, that model scored 0 out of 40. With memory, the same model scored 39 out of 40. Nothing about the model itself changed; the only difference was whether it had memory to work with.
The pattern across all four: memory's value isn't fixed — it scales with how much context would otherwise be lost. On a small problem, there's barely any context to lose, so memory has nothing to offer. On a large one, there's a lot on the line, and memory earns its keep.
Four Key Findings
This benchmark produced four findings — most vendors would only publish one. Here's what memory delivered.
Doesn't improve code quality
Quality ceiling is a property of the model (84-96% across all conditions).
Reduces exploration overhead
28-40% fewer turns, 22-32% lower cost on complex tasks.
Only helps past a threshold
On trivial codebases, memory is pure overhead.
Enables smaller models
Haiku 4.5 scored 0/40 without memory, 39/40 with it. Memory isn't just efficiency — it's capability for smaller models.
What Everyone Else Claims
Industry claims vs. what they actually measure.
| System | Claimed Saving | Actually Measures | Benchmark |
|---|---|---|---|
| Mem0 | 90% tokens | Memory compression | LOCOMO |
| A-Mem | 85-93% tokens | Per-operation cost | Dialogue QA |
| MemMachine | 80% tokens | Recall accuracy | LOCOMO |
| Zep | 94.8% accuracy | Retrieval precision | Custom |
| Letta | N/A (honest) | Agent capability | Terminal-Bench |
| Stompy | 15-28% | Task efficiency | Coding tasks |
Methodology
Worth being specific about what was measured, because “we benchmarked it” is the least falsifiable sentence in software.
- System: MCP-based, PostgreSQL, VoyageAI embeddings
- Codebase: 4,895 lines Python/FastAPI, 158 source files
- Three conditions: stompy (MCP recall), file (static CONTEXT.md), nomemory (cold start)
- Three tasks of increasing complexity
- Scoring: 25-point rubric (5 criteria x 5 points)
- All runs: Claude Opus 4.6, identical codebase snapshot
Results
Five tables: the per-task and aggregate scores from the production-codebase run, the complexity gradient that shows the result moving rather than sitting still, and the swarm results and cost savings from Phase 3.
Table 4 — Per-Task Results
| Task | Stompy | File | NoMemory |
|---|---|---|---|
| Task 1 (Moderate) | 23/25 $1.90 35t | 23/25 $1.80 29t | 24/25 $1.17 19t |
| Task 2 (High) | 21/25 $1.33 31t | 22/25 $3.22 51t | 21/25 $3.51 58t |
| Task 3 (Very High) | 23/25 $3.52 47t | 23/25 $3.16 44t | 22/25 $3.18 54t |
Table 5 — Aggregate
| Condition | Quality | Cost | Turns | Cost/Point |
|---|---|---|---|---|
| Stompy | 67/75 | $6.75 | 113 | $0.101/pt |
| File | 68/75 | $8.18 | 124 | $0.120/pt |
| NoMemory | 67/75 | $7.86 | 131 | $0.117/pt |
Table 6 — Complexity Gradient
| Task | Winner | Turn Savings | Cost Savings |
|---|---|---|---|
| Task 1 (Moderate) | nomemory | -35% turns | -20% cost |
| Task 2 (High) | stompy | +28% fewer turns | +22% cost savings |
| Task 3 (Very High) | stompy | +18% fewer turns | +22% cost savings, +32% time savings |
Tables 7 and 8 are newer than the downloadable report below, which covers Phases 1–2 and lists swarm testing as future work. These Phase 3 results aren't in that PDF yet.
Table 7 — Phase 3: Multi-Agent Swarm Results (6 agents, full-stack booking feature)
| Model | Condition | Score | Cost | Turns | Time |
|---|---|---|---|---|---|
| Sonnet 4.6 | stompy | 40/40 | $3.98 | 2 | 6.5m |
| Sonnet 4.6 | nomemory | 40/40 | $7.04 | 4 | 9.6m |
| Opus 4.6 | stompy | 40/40 | $4.34 | 29 | 9.6m |
| Opus 4.6 | nomemory | 40/40 | $7.65 | 70 | 10.0m |
| Haiku 4.5 | stompy | 39/40 | $4.95 | 2 | 7.5m |
| Haiku 4.5 | nomemory | 0/40 | $3.97 | 3 | 5.8m |
Table 8 — Phase 3: Cost Savings Summary
| Model | With Memory | Without | Savings |
|---|---|---|---|
| Sonnet 4.6 | $3.98 | $7.04 | 43% |
| Opus 4.6 | $4.34 | $7.65 | 43% |
| Haiku 4.5 | $4.95 (39/40) | $3.97 (0/40) | Memory enables capability |
When Memory Hurts
The first benchmark failed. A page that hides this isn't worth reading.
Phase 1: Toy Codebase Results
nomemory won: 70.3% quality vs 59.5% for stompy.
We're showing this because cherrypicking is dishonest.
There is a complexity threshold below which memory is pure overhead. On a trivial, roughly 200-line Express/TypeScript codebase, the model can hold the entire context in its window. Memory retrieval adds latency and noise without reducing exploration — because there is nothing to explore. The breakeven point appears to be around 2,000-3,000 lines of meaningful code with non-obvious architecture.
Use Cases
Multi-Agent Swarms
Problem: 6 agents independently explore codebase. 6x redundant exploration.
How Stompy helps: Lead locks architecture, workers recall. 43% cost savings (Sonnet & Opus). Haiku: 0/40 → 39/40 with memory.
# Lead agent stores architecture
lock_context("service_layer: PostgreSQLAdapter pattern, execute_query for reads, execute_update for writes...")
# Worker agents recall before coding
recall_context("my_app/service_layer") # deeplink syntaxTicketing & Project Management
Problem: Re-explain ticket schema every session.
How Stompy helps: Lock conventions once. All ticket tools benefit.
lock_context("ticketing_workflow: states=[open,in_progress,review,done], priorities=[critical,high,medium,low]...")
ticket(action="create", title="Fix auth bug", type="bug")
ticket_board(status="in_progress")Admin & Operations
Problem: Infra decisions in Slack/heads. 3am incidents.
How Stompy helps: Lock operational knowledge.
lock_context("deployment: DO App Platform, NYC region, auto-deploy on main...")
recall_context("my_app/deployment")
db_query("SELECT * FROM mcp_global.mcp_sessions WHERE status = 'active'")Cross-Session Development
Problem: Monday morning, explain project for 14th time.
How Stompy helps: Stompy remembers across sessions.
lock_context("api_conventions: REST endpoints at /api/v1/, Pydantic models...")
recall_context("my_app/api_conventions")
context_search("how do we handle auth")Codebase Onboarding
Problem: New dev asks same questions previous dev already answered.
How Stompy helps: Previous dev's knowledge persists.
project_brief()
recall_batch(topics=["my_app/service_layer", "my_app/database_conventions", "my_app/test_patterns"])Limitations
The honest gap is scale: these are controlled runs, not a fleet. N=1 per cell in the swarm phase is a promising signal, not statistical proof — better said outright than have you try to work it out. The most valuable numbers come from beta users running this on their own codebases, not on Stompy's.
- Three models (Opus 4.6, Sonnet 4.6, Haiku 4.5), single codebase, N=1 per cell
- Our own system on our own codebase
- Pilot study, not statistical proof
What's Next
The next question isn't whether memory helps — it's how far the pattern holds.
Completed
- Phase 3: Multi-agent swarm — 43% cost savings across Opus & Sonnet, memory enables Haiku (0/40 → 39/40)
Upcoming
- Multi-model validation (GPT-5-Codex, Gemini 2.5 Pro)
- TOON serialization format efficiency
- Longitudinal study: 27 sessions on sustained development
We built a memory system. We tested it honestly. The results were modest. We think modesty, grounded in controlled measurement, is what this field needs.
Be one of the first to use AI memory that goes wherever you go. Beta's open.