AdaMAST

Learn how your AI agents fail, from their own recorded work.

AI agents (coding assistants, tool-using pipelines, multi-agent systems) don't fail randomly. Each system tends to fail in its own recurring, recognizable ways: the checker that always waves work through, the plan that quietly drops a requirement, the tool result that gets ignored. Most teams have no good way to name those patterns, count them, or watch them change.

AdaMAST reads the logs of your agent's past runs and automatically builds a catalog of that system's failure patterns (we call it a taxonomy), with every entry backed by real quotes from your own logs. You can then use the catalog to grade new runs, spot regressions, and feed improvement loops with what went wrong and why instead of just a score.

• 📄 Works on the logs you already have. Common agent log formats are auto-detected
• 🔍 Every failure pattern comes with evidence. Verbatim quotes from real runs
• ✅ Catalogs are quality-gated. Several independent automated reviews must agree before one is accepted
• 🔌 Live mode. Plug into Codex or Claude Code and the catalog is learned and applied while you work

Paper: Fantastic Adaptive Taxonomies and How to Use Them · Website: AdaMAST · Blog: AdaMAST announcement · Docs: Reference

🧪 How it works

traces → independent annotators → agreement gate → accepted taxonomy → judge new runs

• Propose. Several independent automated annotators read your traces, and each proposes failure patterns on its own.
• Agree. The proposals are reconciled. A catalog is accepted only when the independent annotations agree with each other; otherwise it is redrafted. (The full protocol and its acceptance criteria are in the paper.)
• Apply. Judge new runs against the accepted catalog: each trace gets its best-matching failure code, with verbatim evidence quoted from the run.

Every entry in the catalog belongs to one of three categories:

💡 Use cases

📦 Install

Running AdaMAST live inside your coding agent? Install the native plugin; nothing needs to be set up first:

/plugin marketplace add multi-agent-systems-failure-taxonomy/AdaMAST
/plugin install adamast@adamast

codex plugin marketplace add multi-agent-systems-failure-taxonomy/AdaMAST
codex plugin add adamast@adamast

The first block is Claude Code, the second Codex. Both install hooks and the guidance skill, then learn a taxonomy from your own conversations. Details and the package-based alternative are under Runtime integration.

Using the CLI to generate or judge taxonomies from trace files? Requirements: Python 3.10+.

pip install adamast

Verify, with the bundled examples (no model calls):

python -m adamast.examples
adamast validate adamast-examples/traces.jsonl

🚀 Use it

1. Unpack the example files. Every command in this section reads them, and they are written into whichever directory you run this from:

python -m adamast.examples

2. Set one provider credential. OpenAI is the default, so no --provider flag is needed; Anthropic, Google, and AWS Bedrock work the same way with --provider or ADAMAST_PROVIDER (see Providers):

export OPENAI_API_KEY="..."

Generate a taxonomy from a trace file or folder (any of the 7 auto-detected formats):

adamast generate --traces adamast-examples/traces.jsonl --output ./my-taxonomy --view

Judge new traces with it:

adamast judge --taxonomy ./my-taxonomy/taxonomy.json --traces adamast-examples/traces.jsonl --output judgments.json

A ready-made taxonomy ships too, so judging works without waiting on generation:

adamast judge --taxonomy adamast-examples/taxonomy.sample.json --traces adamast-examples/traces.jsonl --output judgments.json

The everyday commands, each runnable as written against the bundled examples:

Only generate and judge call a model; validate, normalize, and view need no credential.

Deeper guides: Trace formats · Generation · The agreement gate · Judging · Judge types · Outputs

🔌 Runtime integration

AdaMAST can also run live inside Codex or Claude Code: hooks checkpoint the agent's work at natural boundaries, record evidence, and learn a project-specific taxonomy automatically from completed conversations. No API key or config is needed for the interactive path. Until your project's own catalog is learned, conversations start from a built-in adaptation of the MAST taxonomy ("Why Do Multi-Agent LLM Systems Fail?", Cemri et al., 2025).

Claude Code

Two paths. Pick one — both register the same hooks, so do not run both.

A · Plugin (recommended). Nothing to install first:

/plugin marketplace add multi-agent-systems-failure-taxonomy/AdaMAST
/plugin install adamast@adamast

The plugin ships the skill, hooks, and taxonomy subagent together. On first use it installs its version-pinned runtime privately, so Python, pip, uv, and the claude CLI do not need to be installed beforehand. See the plugin README.

B · Package CLI. Prefer this if you already manage AdaMAST as a dependency, pin versions, or want a project-local install:

uv tool install adamast          # or: pip install adamast
adamast claude install --user-level

Requires the claude CLI binary on PATH — the installer verifies the hook contract against it and aborts without it.

Codex

A · Plugin (recommended). Nothing to install first:

codex plugin marketplace add multi-agent-systems-failure-taxonomy/AdaMAST
codex plugin add adamast@adamast

Open /hooks in Codex and trust the new plugin hooks.

B · Package CLI. Use this for project-local registration or advanced installer flags:

pip install adamast
adamast codex install --user-level

Both paths install the guidance skill and the same runtime behavior. Do not enable both paths at once.

Use uv tool install, not uvx. Hook commands embed the interpreter path, and uvx resolves to a content-hashed path inside the uv cache that uv cache clean or a version bump invalidates, silently breaking every hook.

Verify the integration

claude plugin list                 # native Claude Code plugin
codex plugin list                  # native Codex plugin
adamast doctor                     # package CLI installation
adamast dashboard --trace-output <program-dir>

Native plugins keep their managed runtime private and do not modify your shell's PATH. Use the plugin list and the host's /hooks view to verify that path; the adamast commands above apply when you installed the package CLI.

Uninstall

/plugin uninstall adamast@adamast          # Claude Code plugin

codex plugin remove adamast@adamast        # Codex plugin
adamast claude uninstall --user-level      # package CLI
adamast codex uninstall --user-level       # package CLI

Learned taxonomies and trace folders are kept so a reinstall resumes; the host guides cover clearing those and dropping the marketplace registration.

The full details (how checkpoints work, the taxonomy picker, background learning, the live monitor, and every knob) live in the runtime integration guide.

📚 Learn more

🤝 Contributing

Development setup, verification commands, and package boundaries: CONTRIBUTING.md

The original research pipeline lives on the paper-pipeline branch; a maintained, locally patched fork is vendored under adamast/learning/vendor/ with provenance in VENDORED.md.

📄 License

Apache-2.0. See LICENSE.