errlore

Memory for AI agents that learns from failures.

Stop the second mistake, not just the first.

Extracted from a 324K LOC production multi-LLM orchestration system, keeping the one part that demonstrably worked: the error-memory loop that made agents stop repeating mistakes.

Your agent keeps making the same mistakes. errlore fixes that:

• Lessons -- every resolved failure becomes a lesson; relevant lessons are injected into the prompt for similar future tasks.
• Known issues -- per-model weakness tracking ("gpt-5.5 keeps hallucinating dates in extraction tasks") injected as warnings.
• Trust -- Bayesian per-model, per-domain trust weights: know which model to pick for which job, based on observed outcomes.
• Closed loop -- errlore tracks whether an injected lesson actually helped and reinforces or decays it automatically.

Embedded, file-based (JSONL), no server, no database, no API keys required. Works fully offline. Your data never leaves your machine.

Quickstart (< 5 minutes)

pip install errlore

from errlore import AgentMemory

mem = AgentMemory("./agent_memory")

# 1. Agent failed -- record it
err_id = mem.log_error("gpt-5.5", "extraction", error="hallucinated dates")

# 2. You fixed it -- extract a lesson
mem.resolve(err_id, "Added date format validation",
            lesson="For date extraction, demand ISO-8601 and verify against source")

# 3. Next similar task -- lessons + known issues injected automatically
inj = mem.inject_for("extract dates from contract", model="gpt-5.5",
                      task_type="extraction")
prompt = f"Your task: extract dates\n{inj.text}"
print(prompt)

# 4. Close the loop -- did the lesson help?
mem.report_outcome(inj, success=True)

# 5. Check stats
print(mem.stats())
# {'errors_total': 1, 'errors_resolved': 1, 'errors_unresolved': 0,
#  'lessons_total': 1, 'lessons_applied': 1, 'pending_injections': 0,
#  'trust': {'gpt-5.5': 0.5522...}}

No API keys needed. errlore itself never calls any LLM -- it manages local JSONL files and does text matching. LLM calls are yours to make (or not).

Does it actually reduce errors?

Yes -- for the class of errors memory can fix. Paired A/B benchmark (benchmarks/bench_error_reduction.py): the same model (claude-haiku-4-5) runs 96 tasks twice, with and without errlore injection. Deterministic validators, no LLM judges; raw outputs committed in benchmarks/results/error_reduction/.

Exact McNemar over all 96 pairs: p = 1.8e-09 (49 pairs fixed, 6 broken). Split by error class:

• Knowledge-gap errors (workspace conventions: date formats, ID normalization, rounding rules, CSV column order): 46/48 -> 0/48, a 100% reduction. The model didn't know the convention; a lesson told it.
• Capability-gap errors (letter counting, string reversal): 17/48 -> 20/48 -- errlore did not help and slightly hurt. Memory fixes what the model doesn't know, not what it can't do.

Reproduce: python benchmarks/bench_error_reduction.py (needs an Anthropic API key; the task families and validators ship in the repo).

How it works

errlore runs three reinforcement loops around your agent:

1. Lesson loop

Agent fails  -->  log_error()  -->  resolve() + lesson
                                        |
Agent runs   <--  inject_for() <--------+
     |
     +--> report_outcome(success=True)  -->  lesson confidence +0.1
     +--> report_outcome(success=False) -->  lesson confidence -0.1

Lessons with high confidence surface first. Unused lessons decay over time.

2. Known-issue loop

Per-model, per-task-type error tracking. When a model has failed on a task type before, inject_for adds a warning block to the prompt. Separate from lessons: lessons are solutions, known issues are warnings.

3. Trust loop

Bayesian per-model weights with adaptive learning rate, cold-start blending, entropy enforcement, and temporal decay. After enough observations, call mem.best_model("code_generation") to pick the model that historically performs best on that domain.

Semantic retrieval (optional)

By default, errlore finds relevant lessons via word overlap (zero dependencies). For higher recall on paraphrased queries, enable embedding search:

pip install errlore[embeddings]   # installs fastembed + numpy

mem = AgentMemory("./agent_memory", embeddings=True)

Benchmark (adversarial paraphrasing)

Tested on 40 lessons with adversarially paraphrased queries (benchmarks/bench_retrieval.py):

The gold set is intentionally adversarial (queries share few literal words with the lesson text), which is why word-overlap scores zero. On natural queries with shared vocabulary, word-overlap works fine.

Integrations

errlore is framework-agnostic. It produces a text block; you put it in the system prompt.

All examples run offline with python examples/<name>.py (mock responses, no API keys). Set use_api=True to call real models.

API overview

The main entry point is AgentMemory. All other classes are internal -- you only need them for advanced use.

Supporting classes (advanced)

Data & privacy

• All data is stored in local JSONL files in the directory you specify.
• Nothing is sent to any server. errlore itself makes zero network calls.
• Works fully offline -- no API keys, no accounts, no telemetry.
• Files: errors.jsonl, lessons.jsonl, injections.jsonl, trust.json, model_accuracy.jsonl.
• Sidecar files (auto-managed): *.idx (byte-offset index), *.lock (filelock), vectors.npy (embedding vectors), vector_meta.json (embedding metadata), trust.json (trust engine state).

Roadmap

• Log compaction for injections journal
• Async API (alog_error, ainject_for, etc.)
• Multi-agent shared memory (multiple agents, one lesson store)
• Lesson clustering and auto-summarization
• Dashboard / CLI for browsing lessons and trust weights
• Export/import for lesson sharing between projects

License

MIT