attemory 0.1.0

Attention-native memory retrieval for local LLM applications.

Retrieval by attending. Search with reasoning.

Attemory is an attention-native retrieval engine that turns large corpora into model-readable memory and reusable KV cache. It retrieves by letting a model attend directly over that memory, delivering high-recall retrieval at scale, validated by SOTA-class results across several benchmarks.

Because retrieval runs through the model's attention path, Attemory can find evidence by reasoning over meaning, constraints, entities, and context, rather than relying on keyword matching or embedding similarity alone. The same mechanism works across conversations, documents, codebases, and long-term memory.

Attemory makes this practical at million-token scale. With partial prefill, KV-cache reuse, and decode-free ranking, it avoids token-by-token generation and turns massive memory into compact, AI-agent-ready evidence sets.

Capabilities

Capability	What it enables
Search with reasoning	Retrieval can follow constraints, combine clues, use dates, names, and infer what memory would answer the real question. Prompts can guide Attemory toward different retrieval behaviors.
Universal retrieval	One system works for conversations, long-term memory, codebases, and any domain where LLMs work.
Million-token scalability	Million-token memories become compact evidence sets that downstream agents can actually use.
Unified search	Exact lookup, fuzzy lookup, entity relationships, and task context live in one context/query format and can be retrieved through natural language. No separate keyword, vector, and graph stack is required.

Benchmark Results

Attemory reaches SOTA-class results across several benchmarks with no benchmark-specific hacks: no query rewrite, no summarization, no agent-driven exploration, and no external cloud services for retrieval. Only the raw corpus and raw benchmark query are used to run the benchmarks.

Benchmark	What it tests	Context size	Attemory result
LongMemEval-S	memory retrieval, the split most memory systems evaluate	about 40 sessions / 115k tokens	98.72% session Recall_any@5, 92.77% session Recall_all@5, 98.94% message Recall_all@50
LongMemEval-M	Million-token memory retrieval, a scale few memory systems attempt	about 500 sessions / 1.5M tokens / 5k messages	94.89% session Recall_any@5, 83.62% session Recall_all@5, 92.55% message Recall_all@50
LoCoMo	End-to-end long-conversation QA	10 long conversations / 1,540 QA items	94.52% accuracy with GPT-4.1-mini as answer model and GPT-4o-mini as judge
Semble	Code retrieval	63 repos / 19 languages / largest repo about 5M tokens	0.9055 file-level NDCG@10

The LongMemEval-M message-level result is the clearest signal of Attemory's capability: Attemory searches roughly 1.5M tokens and 5k historical messages per query, then retrieves all labeled evidence messages within the top 50 messages for 92.55% of answerable queries. This is the retrieval ability the agentic era needs: turning massive memory into compact, exact, actionable evidence.

Semble shows the same idea outside chat memory. Attemory indexes code as raw code chunks and retrieves at chunk level, then maps evidence back to files. On the largest repository in the run, zig, it searches about 5M indexed tokens and reaches 0.9565 file-level NDCG@10.

All benchmarks are reproducible in a local environment. See benchmarks/ for detailed results and run instructions.

Technologies

Attemory retrieves through the same core mechanism that made LLMs powerful: attention. Memories are indexed as reusable KV state, so the query can attend over memory context and use the transformer's reasoning path to decide what is relevant.

Partial prefill and decode-free ranking make this practical, and the prefill path is heavily optimized for speed. KV quantization reduces memory and storage cost, while CUDA GPU and Apple Metal backends accelerate indexing and search on local hardware.

Under the hood, Attemory uses Qwen3.5 as the default retrieval model, with model tiers from 0.8B (tiny) to 2B (small), 4B (medium), and 9B (large) for higher retrieval quality.

The New Retrieval Paradigm

Attemory treats retrieval as attention over a structured context template. The template contains the system prompt, memory, query context, and final query; the data to be searched is placed in the memory section, and the model retrieves by attending over the resulting context. See doc/usage.md#the-context-structure for the concrete structure.

While building Attemory, we found two patterns that consistently improve retrieval quality. The first is query repetition with retrieval guidance, a manual CoT-like way to guide the retrieval process: repeat the question in the query context, then add instructions about what the model should focus on before the final query. This creates a more deliberate retrieval path and helps the model rank memories according to the user's actual constraints. The example shows this pattern in practice.

The second pattern is iterative filtering for long contexts. Attemory splits large memory into segments(due to it's limited context size) and searches each segment independently. The results from each segment are then placed back into the same context template and filtered again. This process repeats until the remaining results come from a single segment, which becomes the final retrieval result.

These retrieval patterns are central to Attemory's results on LongMemEval-M and Semble benchmarks, where it delivers SOTA-class performance on million-token memory and large codebases. See benchmarks/README.md for details.

Getting Started

Attemory is under active development, and behavior may change between versions. It currently supports Linux and macOS. Hardware acceleration is available on NVIDIA CUDA and Apple Metal now.

Install Attemory:

uv pip install attemory           # macOS Apple Silicon, includes Metal runtime
uv pip install "attemory[cpu]"    # Linux CPU
uv pip install "attemory[cuda]"   # Linux + NVIDIA GPU

The same install targets work with pip:

pip install attemory
pip install "attemory[cpu]"
pip install "attemory[cuda]"

On macOS Apple Silicon, attemory automatically installs the Metal-capable runtime, which can run both --backend metal and --backend cpu. On Linux, a bare attemory install only installs the Python package; choose cpu or a CUDA extra explicitly. For CUDA runtime wheels, use cuda-cu126 by default. If you are using a Blackwell GPU such as RTX 50 series, use cuda-cu129. Use cuda-cu124 or cuda-cu121 only when your NVIDIA driver is too old for CUDA 12.6.

Start a local server:

attemory-server --small --backend gpu --port 9006
attemory-server --small --backend metal --port 9006
attemory-server --tiny --backend cpu --port 9006

Tier choice depends on hardware and quality requirements. Practical starting points are --tiny for CPU or small local tests, --small for GPUs with about 5 GB VRAM, --medium for about 8 GB VRAM, and --large for about 12 GB VRAM. Actual memory use also depends on context length and KV type.

Persistent session data and KV cache data are stored on disk. By default, session data uses $XDG_DATA_HOME/attemory/sessions or ~/.local/share/attemory/sessions, and KV cache data uses $XDG_CACHE_HOME/attemory or ~/.cache/attemory. Use ATTEMORY_DATA_DIR, ATTEMORY_CACHE_DIR, or --cache-dir when you need explicit storage paths.

For the full usage guide, including Python APIs, CLI commands, server options, model tiers, context templates, and persistence, see doc/usage.md.

Example

The repository includes a small weekly diary example that shows the basic Attemory workflow: create a session, add memories, index the session, and search with natural language.

attemory-server --large --backend gpu --port 9006 &
python examples/weekly_diary.py

from attemory import AttemoryClient, MemoryInput

client = AttemoryClient(host="127.0.0.1", port=9006, session_id="weekly-diary")
client.create_session()

client.add_system(
    "Read the following weekly diary entries carefully and answer the query at the end."
)

# Context lines have no id. They structure the memory without becoming returned evidence.
client.add_memory(MemoryInput(text="[Wednesday]"))

# Raw memories keep user-owned ids. Attemory returns these ids in search results.
client.add_memory(
    MemoryInput(
        id="20",
        text="In the evening, I had dinner with Clara at a Japanese restaurant.",
    )
)

client.index_session()

results = client.search(
    "Who did I have dinner with at Japanese restaurant?",
    top_k=3,
)

for result in results:
    print(result.id, result.text)

Example output with the large tier:

== Direct fact ==
query: Who did I have dinner with at Japanese restaurant?
--- Results ---
rank=1 line=20 text=In the evening, I had dinner with Clara at a Japanese restaurant. ✓
rank=2 line=34 text=In the evening, I had dinner with Emma at a small French bistro.

== Query context: evening social activity ==
query: Who did I meet on Thursday?
query_context: The user is asking about social activities at evening.
--- Results ---
rank=1 line=27 text=In the evening, I had dinner with David at a Korean barbecue restaurant. ✓
rank=2 line=25 text=Later, I met my parents at town center, we had happy lunch together.

== Query context: family communication ==
query: Who did I meet on Thursday?
query_context: The user is asking about family activities at noon.
--- Results ---
rank=1 line=25 text=Later, I met my parents at town center, we had happy lunch together. ✓
rank=2 line=27 text=In the evening, I had dinner with David at a Korean barbecue restaurant.

== Temporal reasoning ==
query: Assume today is Thursday, who did I have dinner with yesterday?
query_context: The user's query: Assume today is Thursday, who did I have dinner with yesterday?
Resolve relative date into target date before ranking diary entries.
--- Results ---
rank=1 line=20 text=In the evening, I had dinner with Clara at a Japanese restaurant. ✓
rank=2 line=13 text=In the evening, I had dinner with Ben at an Italian place.

This is the core Python API: add context and memories, build the index, then retrieve ranked memory ids and text. See examples/weekly_diary.py for a complete runnable version.

Build From Source

Developers building Attemory from source need a C++17 compiler, CMake 3.18 or newer, and an attemory-core SDK. Pass the SDK root explicitly with ATMCORE_SDK;

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DATMCORE_SDK=/path/to/attemory-core-sdk

cmake --build build --target attemory_server --parallel

Future Work

• MCP support for agent and tool integrations.
• Continued performance optimization for indexing, search, and native backends.
• Broader test coverage across APIs, packaging, persistence, and runtime variants.

Acknowledgements

Attemory is built on the work of the Qwen team and the ggml/llama.cpp community.

Citation

If you use Attemory in research or benchmarks, please cite it as:

@software{attemory2026,
  title        = {Attemory: Attention-Native Memory Retrieval System},
  author       = {Lance Fang},
  year         = {2026},
  url          = {TODO},
}

License

Attemory is released under the MIT License. See LICENSE.