lessonweaver

Turn your agent's real failures into reviewed AGENTS.md, Claude, and Copilot skills — deterministically, with a human in the loop.

lessonweaver mines your agent's execution traces for the mistakes it keeps repeating, puts a human review gate in front of every candidate, and exports the approved ones as the instruction formats teams already use: AGENTS.md fragments, Claude skills / rules / CLAUDE.md, GitHub Copilot instructions, Codex skill directories, and generic runtime snippets.

The human-review gate is the point. Unlike letting a model rewrite its own instructions, every skill here is reviewed, governed, and auditable — no context poisoning, no automatic self-training, no skill activation without review. The core is deterministic with no LLM calls. See when not to create a skill.

Before / after

• Before: A coding agent "reviews" a PR from the title and description without inspecting the diff. A human corrects it. The same mistake recurs next week.
• Trace: The run is recorded, including the human_correction event.
• lessonweaver: Detects a conservative candidate, a human reviews it via multiple-choice questions, and approves it into a reviewed skill.
• After: The reviewed lesson is exported as a Markdown/runtime snippet and pasted into AGENTS.md / Copilot / Claude instructions, so future sessions start with "inspect the changed files first."

What it is / what it is not

lessonweaver complements observability, evals, and memory — see comparisons and ecosystem.

How it works

A trace becomes governed guidance only after a human review gate and a governed promotion gate (shaded). Nothing activates automatically.

flowchart LR
    Run[Agent run] --> Trace[Execution trace]
    Trace --> Detect[Detect candidate]
    Detect --> Review{Human review}
    Review -- reject --> Drop[Discarded]
    Review -- approve --> Lesson[Approved lesson]
    Lesson --> Skill[Skill card]
    Skill --> Promote{Governed promotion}
    Promote -- lint passes --> Active[Active skill]
    Active --> Export[Export: AGENTS.md / Claude / Copilot / runtime]
    Export --> Load[Loaded into agent context]
    Load --> Run

    classDef gate fill:#fde68a,stroke:#b45309,color:#1f2937;
    class Review,Promote gate;

The Mermaid source is kept in docs/assets/lessonweaver-lifecycle.mmd. See docs/architecture.md for the module-level data flow.

Quickstart

The first PyPI release is being prepared (#64). Until it lands, install from source:

pip install -e ".[dev]"
lessonweaver --help

Once published, the user install will be pip install lessonweaver (see Contributing for the contributor workflow). The release process is documented in docs/release.md.

# 1. Detect candidates from a trace and save them to a temporary registry
lessonweaver detect examples/traces/github_pr_review_failure.json \
  --save --registry-root /tmp/lw

# 2. Generate review questions
lessonweaver interview trace-gh-pr-review-001-human-correction --registry-root /tmp/lw

# 3. Record a review answer
lessonweaver answer trace-gh-pr-review-001-human-correction decision approve \
  --free-text "Diff inspection is required before review conclusions." \
  --registry-root /tmp/lw

# 4. Approve into an operational lesson and skill
lessonweaver approve trace-gh-pr-review-001-human-correction \
  --approved-by reviewer --registry-root /tmp/lw

# 5. Export the reviewed skill for an instruction surface
lessonweaver export-skill skill-trace-gh-pr-review-001-human-correction \
  --format markdown --registry-root /tmp/lw

Drop --registry-root /tmp/lw to use registry discovery: the LESSONWEAVER_REGISTRY environment variable, then the nearest .lessonweaver/registry/ directory above the current working directory, then ~/.lessonweaver/registry. For full recipes, see the coding-agent cookbook. For a complete worked example with traces, an approved skill, and a validation suite, see examples/coding_agent_pr_review/.

Flagship demo: the closed loop

The examples/closed_loop_contextweaver/ keystone shows the whole loop end to end: a coding-agent failure → detect → human approve → export-skill as a skill card → that card loaded back into an agent's context (via example.py) so the next run starts already knowing not to repeat the mistake. Reproduce the 60-second terminal demo with docs/assets/demo.sh, which runs the detect→review→export steps and ends by loading the reviewed card back into context. (A recorded GIF/asciinema cast of this flow is tracked as a follow-up.)

Runtime loading

from lessonweaver import FileSystemRegistry, SkillLoader

registry = FileSystemRegistry()
loader = SkillLoader(registry=registry)

context = loader.load_for_task(
    task="Review this PR for security issues",
    agent_type="coding",
    tools=["github"],
    scope="project",
    budget_chars=2000,
)
print(context.snippet)

Commands

• lessonweaver detect <trace.json> [--save] [--output FILE] [--dry-run]
• lessonweaver interview <candidate-id-or-json> [--session FILE] [--dry-run]
• lessonweaver resume-interview <session.json> — reload a saved review and print the remaining (adaptive) questions
• lessonweaver answer <candidate-id> <question-id> <option-id> [--free-text ...] [--session FILE]
• lessonweaver approve <candidate-id> [--approved-by ...] [--dry-run]
• lessonweaver export-skill <skill-id-or-json> --format markdown|json|copilot|copilot-repo|copilot-path|claude|claude-skill|claude-rule|claude-md|agents-md|codex|runtime [--applies-to GLOB] [--redact|--no-redact] [--output FILE] [--json] [--dry-run]
• lessonweaver export-lesson <candidate-id-or-json> --format eval|guardrail|workflow [--redact|--no-redact] [--output FILE] [--json] [--dry-run]
  • For a coherent eval/guardrail/workflow bundle, use export_eval_companion_pack from the library API; see eval companion.

Redaction is on by default for export-rendering commands; pass --no-redact when you intentionally need raw content. Shared output flags: --output FILE writes the result to a file instead of stdout; --json wraps export-skill/export-lesson output in a {"format": ..., "content": ...} envelope for scripting; --dry-run previews a command without writing any file or registry entry (it prints a [dry-run] would write to: ... notice when --output is set). For review-trace, default redaction is applied to the full JSON packet, including the optional export_preview generated by --target. The review interview is adaptive: marking a lesson high risk or workflow_change queues a follow-up question, and a reject decision skips the remaining scoping questions. Save progress with interview --session FILE (a registry-backed candidate is required so it can be reloaded), record answers into that session with answer --session FILE, and continue later with resume-interview, which lists the remaining adaptive questions from the answers recorded so far.

Commands return non-zero on bad input and keep stdout clean for JSON consumers. Application-level error lines start with Error: on stderr; warnings start with warning:. Malformed argv rejected by argparse uses argparse's standard usage:/error: output.

• lessonweaver review-trace <trace.json> [--answer q=opt] [--approve] [--target FORMAT] [--redact|--no-redact] [--dry-run] — one guided command for the whole detect→review→(approve) loop (see the developer workflow)
• lessonweaver export-file <skill-id-or-json> --path FILE [--format ...] [--write] [--no-redact] — diff-first, idempotent insertion of a skill into an instruction file (previews a unified diff unless --write)
• lessonweaver lint <skill-id-or-json>
• lessonweaver analyze-skills <skills-dir>
• lessonweaver retrieve "<task>"
• lessonweaver load "<task>" [--explain]
• lessonweaver explain-load "<task>" [--agent-type ...] [--tools ...] — explain which skills load or are skipped (reason codes), budget usage, and overlaps
• lessonweaver cleanup-skills [--write] — report (and optionally apply) cleanup for stale, noisy, and overlapping skills
• lessonweaver validate-skill <suite.json> [--skills-dir DIR | --registry-root ROOT]
  • Suite JSON: {"suite_id": "s1", "skill_id": "pr-review", "examples": [{"example_id": "pos", "task": "Review this pull request", "should_load": true}, {"example_id": "neg", "task": "Generate a SQL migration", "should_load": false}]}. Negative examples (should_load=false) measure precision; the command prints the eval result as JSON and exits 0 when every example passes, 1 otherwise.
• lessonweaver promote-skill <skill-id> <target-status>
• lessonweaver eval-detection benchmark/v1/corpus.json --compare-results benchmark/v1/results.json — reproduce the public detection benchmark scorecard and fail if the recorded results drift.

Closed-loop effectiveness reports are available through the library API; see closed-loop effectiveness.

Supported outputs and integrations

Governance and safety

• Detection is conservative; false negatives are preferred over noisy guidance.
• Human review is the enforced governance step before a lesson becomes an approved skill or exportable lesson: approve, review-trace --approve, and export-lesson refuse to run until the required (adaptive) review questions are answered. --allow-incomplete-review overrides the gate and records the bypass in metadata.
• experimental skills must pass governed lifecycle checks (lint with no errors) before becoming active.
• SimpleRedactor and TraceSanitizer share the same best-effort redaction rules before export and pre-mining sanitization. Redaction markers identify the matching rule, for example [REDACTED by email]. This is a safety net, not a compliance control.
• Skills carry owner, approver, expiration, sensitivity, scope, and evidence metadata.

See when not to create a skill — turning every observation into a skill causes context poisoning.

Anti-goals

• No LLM-based lesson generation in the core library.
• No automatic skill injection without human approval.
• No agent orchestration or framework lock-in.
• No replacement for evals, tests, or review gates.
• No compliance-grade privacy scanner.

Roadmap

Grouped by adoption path (tracking issues):

• Operational memory: #57
• Runtime lesson retrieval API: #59
• Eval companion: #60
• Closed-loop effectiveness measurement: #61
• Policy-gated lesson promotion: #62

Part of the Weaver Stack

lessonweaver is the learning loop of the Weaver Stack — the part that closes the loop. Sibling tools produce evidence (agent-kernel ActionTraces, ChainWeaver flow failures, vibeguard findings), lessonweaver turns that evidence into reviewed, governed guidance, and that guidance feeds back into agents (for example, contextweaver can load exported skill cards, or you paste them into AGENTS.md).

flowchart LR
    subgraph Producers[Siblings produce evidence]
        AK[agent-kernel<br/>ActionTraces]
        CW[ChainWeaver<br/>flow failures]
        VG[vibeguard<br/>findings]
    end
    Producers -->|traces / findings| LW[lessonweaver<br/>detect to review to export]
    LW -->|reviewed skill cards| Consumers
    subgraph Consumers[Guidance feeds agents]
        CX[contextweaver<br/>skill-card loader]
        AM[AGENTS.md / Claude / Copilot]
    end
    Consumers --> Better[Better agents]
    Better -.->|next run| Producers

    classDef hub fill:#bfdbfe,stroke:#1d4ed8,color:#1f2937;
    class LW hub;

lessonweaver works standalone — it consumes any trace in the documented trace format and takes no hard runtime dependency on any sibling. The Mermaid source is in docs/assets/lessonweaver-closed-loop.mmd.

Documentation

• Architecture — modules, data flow, lifecycle
• Glossary — canonical terms
• Comparisons — vs. observability, evals, memory, frameworks
• Ecosystem positioning — integration boundaries
• When not to create a skill
• Developer workflow — guided review, diff-first export, load diagnostics, cleanup
• Coding-agent cookbook
• Repository-check findings cookbook
• Integrations: LlamaIndex, OpenAI Agents SDK, Pipecat
• Interoperability
• Adapters & trace import contract — the TraceImporter protocol
• Trace format
• Detection benchmark and contribution guide
• Repository readiness
• Examples
• Agent instructions

Contributing

See CONTRIBUTING.md for principles, local development, and good first issues. By participating you agree to the Code of Conduct. Notable changes are tracked in the changelog, and security reports follow the security policy.

pip install -e ".[dev]"
ruff check src/ tests/
ruff format --check src/ tests/
mypy src/lessonweaver/
pytest