Like Johnny, your agent carries more than its head can hold — and keeps working for hours.
A real coding session — a canvas tower-defense game built from scratch by a 35B-A3B MoE model on a single 12 GB RTX 4070 Ti — with the proxy in between. Nothing was summarized away: every message, tool call and output stays on disk, verbatim, and the model can pull any of it back.
Developed and benchmarked primarily with Strata, by Niko1221 and contributors. mnemonic-proxy is an independently maintained project that also supports llama-server, ds4-server and other OpenAI-compatible backends; saved engine state depends on what the backend supports.
Replay of a real exchange (GIF). Three hours and two segment switches in, one recall call — resolved inside the proxy in 61 ms — brings back the verbatim first version of the boss block, message 20. No model summarized anything.
Magenta: the full history of the session. Cyan: what the model actually reads on each request. Dashed lines: the two segment switches (≈2 minutes each, written handoff notes). Data from the proxy journal, 5 Oct 2026.
An OpenAI / Anthropic / Responses-compatible HTTP proxy. Your agent talks to it exactly as it would talk to the model. No summaries written by a model, no vector database required.
Old tool outputs and reasoning leave the prompt in stable blocks and become one-line receipts with an id. The prefix stays byte-identical between blocks, so the engine's KV cache keeps working.
A recall tool searches the verbatim archive: first write of a file, its full edit timeline, command → output, the reasoning behind an edit — every hit tagged with message number and segment.
When even the masked window fills up, the model writes its own notes, the segment is sealed and a fresh one starts. Pinned lines (!!) survive every switch.
On engines that support it (llama-server, Strata) the KV state is saved to disk: resuming after a pause, a restart or another client takes 1–3 s instead of a full re-read.
Old state files are rebuilt bit-for-bit from content-addressed zstd blocks: −36% disk on our sessions, sha256-verified before every restore.
Agents ship tens of thousands of tokens of tool definitions. The proxy keeps the core ones and loads the rest by name: 25.6K → 6.2K tokens on every request.
"In the courier business you don't read the payload. You carry it, and you give it back exactly as it was."— the design rule. Hidden content is never paraphrased; it is either in the prompt or retrievable verbatim.
71 questions about past sessions, each with the known location of the evidence (including 8 with no answer in the archive). Held-out split:
| keyword search | structured recall | |
|---|---|---|
| evidence found within 5 calls | 90% | 100% |
| tokens returned per question | 5,940 | 2,737 |
| "not in the archive" recognised | 1 / 4 | 4 / 4 |
The benchmark guided development, so read it as a regression suite, not an independent score.
Resuming a conversation instead of re-reading it:
| case | saved | re-read |
|---|---|---|
| Strata (session-files branch), 110K tokens, RTX 4070 Ti | 1.4 s | ≈50 s |
| Strata (session-files branch), 69K tokens, after restart | 2.9 s | 33 s |
| llama-server, 8.2K tokens, CPU, Qwen3-0.6B | 1.9 s | 28.6 s |
Context restore only; loading the model is separate.
| on Strata, 12 GB GPU | Claude Code | Codex |
|---|---|---|
| session | 1 h 41 min, 339 requests, 0 proxy errors | 20 requests |
| prompt tokens from the engine cache | 92.6% | 93.9% |
| context | 328K virtual, ≤46K in the prompt, 14 segment switches | exact recall: 36 tests, 8/8 messages |
Claude Code 2.1.284 on a low-threshold test config (masking at 36K) so that segments switch. Facts scattered in 16K–122K prompts on Strata alone: 151/160 exact, 0 wrong, 9 omitted (all ≥ 64K).
Why it matters. Without saved state, the whole prefill of a long conversation is paid again every time the engine loses it — after a restart, when another client used the engine in between, when the proxy returns to a sealed segment. On a 12 GB GPU that is close to a minute for 100K tokens; restoring takes a second or two, from disk, not VRAM. Any client of a local engine benefits, not just this proxy. llama-server has it (--slot-save-path); for Strata it is proposed in PR #668, not yet merged.
Python ≥ 3.10, standard library only (optional: tokenizers for exact counts). Point it at any OpenAI-compatible engine; with llama-server you also get saved state. Engine window: at least 16K, 32K or more recommended — below 16K the proxy refuses to start.
# 1. an engine — llama.cpp with slot saving $ llama-server -m model.gguf -c 131072 --slot-save-path ./slots --port 8095 # 2. the proxy $ pip install mnemonic-proxy # or: uvx mnemonic-proxy demo $ mnemonic-proxy --upstream http://127.0.0.1:8095 --port 8096 --data ./data # 3. your agent → http://127.0.0.1:8096 $ OPENAI_BASE_URL=http://127.0.0.1:8096/v1 pi # Chat Completions $ ANTHROPIC_BASE_URL=http://127.0.0.1:8096 claude # Anthropic Messages $ codex -c model_providers.local.base_url=http://127.0.0.1:8096/v1 \ -c model_providers.local.wire_api=responses # Responses
| client \ engine | Strata | llama-server | ds4-server | any OpenAI-compatible |
|---|---|---|---|---|
| Chat Completions (pi, Hermes, Continue…) | tested · full | tested on CPU · full | tested with pi · engine keeps its own state | base (no saved state) |
| Claude Code (Anthropic Messages) | tested live | experimental | experimental | experimental |
| Codex (Responses) | tested live | experimental | experimental | experimental |
| MCP recall server (Claude Code, Codex, Hermes) | tested read-only on a real archive | |||
ds4-server (antirez/ds4) with pi on a Ryzen AI Max+ 395, 8K window: GLM 5.3 Flash Q2 recalled an error line removed from the prompt (right id and shard, plus an invented timestamp), and after each recall ds4 read only the new 614–634 tokens. The 8K window was chosen to stress recall; the proxy now requires at least 16K. mnemonic-proxy is a separate tool, compatible with ds4-server; it is not part of the ds4 project.
--slot-save-path. Our numbers were measured on the 0.1.38 + PR #668 build; 0.1.40 not yet re-measured with the proxy.NameError). 0.3.0 blocks these writes; tested on a mock engine, not yet live.