AdaMAST

Learn how your AI agents fail, from their own recorded work.

AI agents (coding assistants, tool-using pipelines, multi-agent systems) don't fail randomly. Each system tends to fail in its own recurring, recognizable ways: the checker that always waves work through, the plan that quietly drops a requirement, the tool result that gets ignored. Most teams have no good way to name those patterns, count them, or watch them change.

AdaMAST reads the logs of your agent's past runs and automatically builds a catalog of that system's failure patterns (we call it a taxonomy), with every entry backed by real quotes from your own logs. You can then use the catalog to grade new runs, spot regressions, and feed improvement loops with what went wrong and why instead of just a score.

• 📄 Works on the logs you already have. Common agent log formats are auto-detected
• 🔍 Every failure pattern comes with evidence. Verbatim quotes from real runs
• ✅ Catalogs are quality-gated. Several independent automated reviews must agree before one is accepted
• 🔌 Live mode. Plug into Codex or Claude Code and the catalog is learned and applied while you work

Paper: Fantastic Adaptive Taxonomies and How to Use Them · Website: AdaMAST · Blog: AdaMAST announcement · Docs: Reference

---

🧪 How it works

traces → independent annotators → agreement gate → accepted taxonomy → judge new runs

• Propose. Several independent automated annotators read your traces, and each proposes failure patterns on its own.
• Agree. The proposals are reconciled. A catalog is accepted only when the independent annotations agree with each other; otherwise it is redrafted. (The full protocol and its acceptance criteria are in the paper.)
• Apply. Judge new runs against the accepted catalog: each trace gets its best-matching failure code, with verbatim evidence quoted from the run.

Every entry in the catalog belongs to one of three categories:

Category	Scope	Example
⚙️ System-level	Can arise in any agent system	Context exhaustion
🎭 Role-specific	Tied to a discovered component role	Checker rubber-stamps solver output
🧪 Domain-specific	Requires task knowledge	Algorithm mismatch

💡 Use cases

Scenario	How
🔬 Error analysis: learn what your agent actually gets wrong, with supporting evidence	adamast generate on a batch of traces, then read the field guide
📈 Regression tracking: watch failure patterns across agent versions	adamast judge new runs against the same catalog and compare
🏅 Best-of-N selection: pick the cleanest of several candidate runs	Judge each candidate; prefer the one with the fewest, least severe codes
🧬 Feedback for optimization loops: tell a prompt or agent optimizer why runs failed, not just the score	Feed the judged codes back as the improvement signal
🔌 Live runtime integration: the catalog is learned and applied while you work in Codex or Claude Code	The one that needs setup; see Runtime integration

📦 Install

Running AdaMAST live inside your coding agent? Install the native plugin; nothing needs to be set up first:

/plugin marketplace add multi-agent-systems-failure-taxonomy/AdaMAST
/plugin install adamast@adamast

codex plugin marketplace add multi-agent-systems-failure-taxonomy/AdaMAST
codex plugin add adamast@adamast

The first block is Claude Code, the second Codex. Both install hooks and the guidance skill, then learn a taxonomy from your own conversations. Details and the package-based alternative are under Runtime integration.

Using the CLI to generate or judge taxonomies from trace files? Requirements: Python 3.10+.

pip install adamast

Verify, with the bundled examples (no model calls):

python -m adamast.examples
adamast validate adamast-examples/traces.jsonl

🚀 Use it

1. Unpack the example files. Every command in this section reads them, and they are written into whichever directory you run this from:

python -m adamast.examples

2. Set one provider credential. OpenAI is the default, so no --provider flag is needed; Anthropic, Google, and AWS Bedrock work the same way with --provider or ADAMAST_PROVIDER (see Providers):

export OPENAI_API_KEY="..."

Generate a taxonomy from a trace file or folder (any of the 7 auto-detected formats):

adamast generate --traces adamast-examples/traces.jsonl --output ./my-taxonomy --view

Judge new traces with it:

adamast judge --taxonomy ./my-taxonomy/taxonomy.json --traces adamast-examples/traces.jsonl --output judgments.json

A ready-made taxonomy ships too, so judging works without waiting on generation:

adamast judge --taxonomy adamast-examples/taxonomy.sample.json --traces adamast-examples/traces.jsonl --output judgments.json

The everyday commands, each runnable as written against the bundled examples:

Command	Purpose
adamast validate adamast-examples/traces.jsonl	Check trace files: count, detected formats, empty trajectories
adamast normalize adamast-examples/traces.jsonl --output out.jsonl	Convert any accepted format to canonical AdaMAST JSONL
adamast generate --traces adamast-examples/traces.jsonl --output ./my-taxonomy	Agreement-gated taxonomy generation
adamast judge --taxonomy adamast-examples/taxonomy.sample.json --traces adamast-examples/traces.jsonl	Every supported failure code per trace, with evidence
adamast view adamast-examples/taxonomy.sample.json	Open a taxonomy as a read-only browser field guide

Only generate and judge call a model; validate, normalize, and view need no credential.

Deeper guides: Trace formats · Generation · The agreement gate · Judging · Judge types · Outputs

🔌 Runtime integration

AdaMAST can also run live inside Codex or Claude Code: hooks checkpoint the agent's work at natural boundaries, record evidence, and learn a project-specific taxonomy automatically from completed conversations. No API key or config is needed for the interactive path. Until your project's own catalog is learned, conversations start from a built-in adaptation of the MAST taxonomy ("Why Do Multi-Agent LLM Systems Fail?", Cemri et al., 2025).

Claude Code

Two paths. Pick one — both register the same hooks, so do not run both.

A · Plugin (recommended). Nothing to install first:

/plugin marketplace add multi-agent-systems-failure-taxonomy/AdaMAST
/plugin install adamast@adamast

The plugin ships the skill, hooks, and taxonomy subagent together. On first use it installs its version-pinned runtime privately, so Python, pip, uv, and the claude CLI do not need to be installed beforehand. See the plugin README.

B · Package CLI. Prefer this if you already manage AdaMAST as a dependency, pin versions, or want a project-local install:

uv tool install adamast          # or: pip install adamast
adamast claude install --user-level

Requires the claude CLI binary on PATH — the installer verifies the hook contract against it and aborts without it.

Codex

A · Plugin (recommended). Nothing to install first:

codex plugin marketplace add multi-agent-systems-failure-taxonomy/AdaMAST
codex plugin add adamast@adamast

Open /hooks in Codex and trust the new plugin hooks.

B · Package CLI. Use this for project-local registration or advanced installer flags:

pip install adamast
adamast codex install --user-level

Both paths install the guidance skill and the same runtime behavior. Do not enable both paths at once.

Use uv tool install, not uvx. Hook commands embed the interpreter path, and uvx resolves to a content-hashed path inside the uv cache that uv cache clean or a version bump invalidates, silently breaking every hook.

Verify the integration

claude plugin list                 # native Claude Code plugin
codex plugin list                  # native Codex plugin
adamast doctor                     # package CLI installation
adamast dashboard --trace-output <program-dir>

Native plugins keep their managed runtime private and do not modify your shell's PATH. Use the plugin list and the host's /hooks view to verify that path; the adamast commands above apply when you installed the package CLI.

The full details (how checkpoints work, the taxonomy picker, background learning, the live monitor, and every knob) live in the runtime integration guide.

📚 Learn more

You want to…	Read
Prepare and check trace files	Trace formats
Understand the words (trace, taxonomy, judge, …)	Concepts
Use the Python API instead of the CLI	Runtime API
Fix a broken setup	Troubleshooting
Browse everything	Documentation index

🧰 All commands

Command	Purpose
adamast validate / normalize	Check and convert trace files
adamast generate	Agreement-gated taxonomy generation
adamast judge	Apply a taxonomy to traces
adamast view	Browser field guide for one taxonomy
adamast find	List or select stored taxonomies
adamast import-traces	Generate a taxonomy from existing traces into the local store
adamast register-taxonomy	Register an existing taxonomy file into the local store
adamast doctor	Validate paths, configuration, hooks, and host contracts
adamast status	Active taxonomy, traces, learning state, recent decisions
adamast dashboard	Local taxonomy dashboard / checkpoint monitor
adamast traces	Inspect trace state
adamast claude install / uninstall	Manage Claude Code hooks
adamast claude add-hook / remove-hook / list-hooks	Manage custom Claude Code checkpoint hooks
adamast codex install / uninstall	Manage Codex hooks
adamast claude checkpoint / adamast codex checkpoint	Record a private runtime checkpoint (invoked by the hooks)
adamast single-run	Wrap one direct model task with AdaMAST

🗂️ Repository map

Path	Responsibility
adamast/core/	Taxonomy data model, evidence, traces, taxonomy store/MAST/resolution, session lifecycle
adamast/protocol/	The compact-checkpoint implementation and the pre-submission gate
adamast/judges/	Taxonomy and reflection judges, plus the provider-neutral JUDGES contract
adamast/llm/	Model routing, learning calls, and provider transports
adamast/learning/	Taxonomy generation and refinement, learning jobs, and the vendored/ported pipelines
adamast/hosts/	Claude Code, Codex, interactive, and single-LLM host adapters
adamast/dashboard/	Local dashboard, status, taxonomy viewer, and web views
adamast/examples/	Runnable demonstrations (python -m adamast.examples copies them locally)
adamast/cli.py	The umbrella adamast command
tests/	The single test suite (python -m pytest tests)
docs/	User and contributor documentation (index)
scripts/	Repository tooling: docs-site build, public publishing
website/	The static landing page served ahead of the docs
SKILL.md	The Codex skill manifest for AdaMAST

Everything importable lives in the adamast package; the complete ownership rules are in Architecture.

🤝 Contributing

Development setup, verification commands, and package boundaries: CONTRIBUTING.md

The original research pipeline lives on the paper-pipeline branch; a maintained, locally patched fork is vendored under adamast/learning/vendor/ with provenance in VENDORED.md.

📄 License

Apache-2.0. See LICENSE.