agentworkbench 0.2.0

Local-first harness for learning, evaluating, comparing, and optimizing repo-native agents.

Agent Workbench

Agent Workbench (awb) is a local-first Python package and CLI for evaluating and optimizing repo-native coding agents.

Release-Ready Guides

• Codex Agent Guide — exact workflow for a downstream Codex agent using awb to optimize a target agent.
• Release And Laptop Setup — validation, build, publish, and work-laptop installation steps.
• Multi-Turn And Chain Features — conversation, chain, and workflow details.

What It Does

Agent Workbench helps optimizer engineers understand, evaluate, and improve coding agents (Codex, Copilot, Claude Code) that operate within their own repository. The harness:

1. Learns the repo — discovers prompts, tools, skills, plugins, MCPs, and runtime behavior
2. Captures what "better" means — creates behavior briefs defining desired agent behavior
3. Generates evals — creates test cases from briefs
4. Runs experiments — executes agents against eval cases with controlled variables
5. Analyzes results — diagnoses failures, compares candidates, and recommends improvements

Installation

python3 -m venv .venv
. .venv/bin/activate
pip install -e ".[dev]"

awb --help
python -m agentworkbench --help

Quickstart

# Initialize workbench in current repo
awb init

# Discover the repo structure
awb learn

# Create a behavior brief
awb brief start --brief-id my-brief --agent-id my-agent

# Generate evals from an approved brief
awb evals generate --brief-id my-brief --eval-id my-eval

# Run the agent against eval cases
awb run --agent-id my-agent --eval-id my-eval --subset smoke

# Check loop readiness before promotion
awb check --agent-id my-agent --eval-id my-eval --subset full

# Compare against baseline
awb compare --baseline baseline-id --candidate candidate-id

# Diagnose failures
awb diagnose --eval-id my-eval

---

Core Concepts

Agents

An agent spec defines a target agent with:

• Adapter: How to invoke the agent (command, python, provider)
• Prompts: System prompts and prompt variants
• Model: For provider adapters (model name, provider, temperature)
• Tools: External tools the agent can use
• Skills: Internal capabilities (grounding, memory recall, etc.)
• Plugins: Additional capabilities (knowledge base, workflow memory)
• MCPs: Model Context Protocol servers
• Entrypoints: How to run the agent (command, python function, API)

Example Agent Spec

id: groq-support-baseline
adapter: provider
prompts:
  system: target_agent/prompts/system.md
  reviewer: target_agent/prompts/reviewer.md
model:
  name: groq/llama-3.1-8b-instant
  provider: groq
  temperature: 0
tools:
  - name: policy_search
    enabled: true
  - name: incident_lookup
    enabled: true
skills:
  - name: policy_grounding
    enabled: true
  - name: incident_triage
    enabled: true

Behavior Briefs

A behavior brief captures what "better" means for the agent:

• Goal: What the agent should accomplish
• Desired behaviors: Explicit behaviors to encourage
• Worse signals: Behaviors to avoid
• Preferred surfaces: Which tools/skills/MCPs to use
• Workflow scenarios: Multi-step workflows the agent should handle

Creating a Brief

awb brief start --brief-id my-brief --agent-id my-agent
# Follow prompts to define goal, behaviors, etc.

awb brief finalize --brief-id my-brief

Eval Suites

An eval suite contains test cases for evaluating an agent:

• Cases: Individual test cases with input, expected output, and metadata
• Splits: Named subsets (smoke, full, etc.)
• Scorers: Functions that score agent outputs

Eval Case Structure

{
  "id": "case-1",
  "input": "What should I do after a failed benchmark run?",
  "expected": "diagnose before rerun",
  "rubric": ["Answers are grounded and actionable"],
  "tags": ["output", "smoke"],
  "metadata": {
    "process_expectations": {
      "required_tools": ["repo_search"],
      "max_step_count": 3
    },
    "score_weights": {
      "output_quality": 0.7,
      "process_alignment": 0.3
    }
  }
}

Multi-Turn Eval Cases

{
  "id": "support-multiturn-refund",
  "input": "What refund amount should we offer?",
  "turns": [
    {"role": "user", "content": "A customer wants a refund"},
    {"role": "assistant", "content": "I found the customer's account..."}
  ],
  "metadata": {
    "expected_turns": [
      {"step": 0, "expected_tools": ["policy_search"]},
      {"step": 1, "expected_tools": ["customer_memory"]}
    ]
  }
}

---

CLI Reference

Initialization

awb init

Scaffolds workbench files:

awb init                    # Initialize in current directory

Creates .agent-workbench/ with:

• agents/ — Agent specs
• briefs/ — Behavior briefs
• evals/ — Eval suites
• scorers/ — Custom scorers
• workbench.yaml — Workbench configuration

awb learn

Discovers repo structure:

awb learn

Detects:

• Prompt files
• Tool definitions
• Skill implementations
• Plugin code
• MCP configurations
• Entry points

Behavior Briefs

awb brief start

Creates a new behavior brief:

awb brief start --brief-id support-brief --agent-id my-agent --goal "Help users with support tickets"

awb brief draft

Seeds a behavior brief from an agent spec, linked evals, and discovery artifacts:

awb brief draft --brief-id my-brief --agent-id my-agent

awb brief finalize

Finalizes an approved brief:

awb brief finalize --brief-id my-brief

awb brief packet

Generates a coding agent packet:

awb brief packet --brief-id my-brief --agent-id my-agent

The packet now includes Human Checkpoints when the brief is too subjective or under-specified for the package to lock down alone. Those checkpoints are for target-surface confirmation, representative benchmark examples, answer acceptance decisions, or short guidance to the model. They are not prompts for the human to write a gold answer.

Eval Suites

awb evals generate

Generates eval cases from a brief:

awb evals generate --brief-id my-brief --eval-id my-eval

awb evals expand

Appends draft coverage from the linked brief and recent experiment failures, then resets approval so benchmark drift stays explicit:

awb evals expand --eval-id my-eval --experiment my-agent-my-eval-full-20260410112233445566

awb evals approve

Approves an eval suite:

awb evals approve --eval-id my-eval

awb evals review

Shows eval suite details in a human-readable review report:

awb evals review --eval-id my-eval

The review report includes Human Checkpoints Before Approval when cases still need human help to define subjective acceptance rules or workflow contracts. Record those answers with awb human-review answer --decision ... --guidance ... so the package can stop re-asking the same checkpoint.

Human Review

awb human-review answer

Records a human decision or short guidance without asking the human to write an ideal answer:

awb human-review answer \
  --queue-id eval-my-eval \
  --checkpoint-id eval-subjective-acceptance \
  --decision accept_candidate \
  --guidance "Prefer the shorter grounded answer."

Recorded answers are automatically applied back into the relevant brief/eval artifacts so the same checkpoint can disappear from later brief packet, evals review, and diagnose runs.

awb human-review apply

Re-applies stored answers into benchmark artifacts and refreshes the derived review reports:

awb human-review apply --queue-id eval-my-eval
awb human-review apply --checkpoint-id eval-subjective-acceptance --dry-run

Running Experiments

awb run

Executes an agent against eval cases:

awb run --agent-id my-agent --eval-id my-eval --subset smoke
awb run --agent-id my-agent --eval-id my-eval --case-id case-1 --case-id case-2
awb run --agent-id my-agent --eval-id my-eval --tag workflow
awb run --agent-id my-agent --eval-id my-eval --match "refund"
awb run --agent-id my-agent --eval-id my-eval --case-limit 10

awb run chain

Runs cases through multiple agents sequentially:

awb run chain \
  --agent-id analyzer-agent \
  --agent-id router-agent \
  --eval-id my-eval \
  --subset chain

# With step configuration
awb run chain \
  --agent-id agent-a \
  --agent-id agent-b \
  --eval-id my-eval \
  --subset smoke \
  --step-config '{"agent_id": "agent-b", "branch_condition": "prior_score >= 0.5"}'

Chain Handoff Template

Customize how output is passed between chain steps:

awb run chain \
  --agent-id agent-a \
  --agent-id agent-b \
  --chain-handoff-template "Given: {prior_output}\n\nOriginal: {original_input}"

Template variables:

• {prior_output} — Previous agent's output
• {prior_structured_output} — JSON of structured output
• {prior_tool_calls} — JSON of tool calls
• {prior_evidence} — JSON of evidence records
• {prior_trace} — JSON of trace events
• {original_input} — Original case input
• {case_id} — Case ID
• {step} — Current step index
• {chain_context} — Full accumulated context

awb run workflow

Runs routed multi-agent workflows, including fan-out/fan-in graphs and nested chain nodes:

awb run workflow \
  --workflow-file groq-routing-workflow \
  --eval-id workflow-routing-live \
  --subset full

awb run workflow \
  --workflow-file groq-routing-nested-chain \
  --eval-id workflow-routing-live \
  --subset smoke

Workflow nodes can be plain agents or node_type: chain compositions, so a router can hand work to a nested planner-worker-reviewer style lane instead of only a single downstream agent.

Compatibility alias:

awb run-workflow --workflow-file groq-routing-workflow --eval-id workflow-routing-live --subset full

Comparison & Diagnosis

awb check

Runs local loop-readiness checks for long-running Codex optimization loops:

awb check
awb check --agent-id my-agent --eval-id my-eval --subset full
awb check --experiment exp-id
awb check --workflow-file workflow.yaml --eval-id my-eval --subset full
awb check --chain-agent-id planner --chain-agent-id reviewer --eval-id my-eval --subset full
awb check --format text --fail-on none

The command writes JSON and Markdown reports under .agent-workbench/reports/, including check-latest.json and check-latest.md. By default it exits non-zero only for critical findings. Promotion and compare --best include a loop_readiness gate when loop_checks.block_promotion_on_critical is enabled.

Check categories:

• code_structure: large files, long functions/classes, and unclear target surface splits.
• agent_topology: unnecessary multi-agent setups, missing routing/handoff structure, workflow loops, dead ends, and unmatched routes.
• eval_contract: missing rationales for required/forbidden tools, MCPs, prompt keys, workflow steps, and workflow sequences.
• deterministic_fallback: eval case IDs, exact expected strings, hard-coded output maps, and canned final-output fallbacks in target runtime files.
• loop_hygiene: draft evals, benchmark drift, smoke-only decisions, missing promoted baselines, review items, gates, and scorer-audit issues.

awb compare

Compares experiment results:

awb compare --baseline baseline-id --candidate candidate-id
awb compare --best --candidate candidate-id

awb diagnose

Analyzes failures and recommends improvements:

awb diagnose --eval-id my-eval --subset full
awb diagnose --experiment exp-id

Diagnosis reports include an Ask The Human section when a regression or change cannot be trusted without human preference input or benchmark clarification.

awb status

Shows optimization stream status:

awb status --eval-id my-eval
awb status

Artifacts

awb artifacts show

Inspect stored experiment, case, or compare artifacts:

awb artifacts show experiment --experiment exp-id
awb artifacts show case --experiment exp-id --case-id case-1
awb artifacts show compare --latest

awb artifacts export

Export a full experiment artifact bundle:

awb artifacts export --experiment exp-id --output exp-id.json

awb artifacts review

Generate an HTML review bundle:

awb artifacts review --experiment exp-id --output review.html

Legacy aliases still work for compatibility: run-chain, run-workflow, replay, show, export, review.

Red Teaming

awb redteam generate

Builds adversarial, robustness, or stress suites from an approved source eval:

awb redteam generate \
  --source-eval-id provider-tool-interface \
  --output-eval-id provider-tool-interface-redteam \
  --mode adversarial \
  --mode robustness \
  --mode stress

Generated suites carry attack metadata for family, severity, mode, and source-case lineage, and they include mode-specific splits alongside smoke and full.

awb redteam report

Summarizes outcomes for a completed red-team experiment:

awb redteam report --experiment <experiment-id>

Optimization

awb propose

Generates mutation proposals:

awb propose --eval-id my-eval --subset full

awb optimize

Runs a bounded optimization loop over the existing search primitives, then writes a single artifact that includes the measured frontier, finalist revalidation on the preferred subset, promotion-style gate checks, and a proposal-to-patch handoff:

awb optimize --agent-id my-agent --eval-id my-eval --subset full --budget 6

You can also add an optional model/runtime axis:

awb optimize --agent-id my-agent --eval-id my-eval --subset smoke --budget 8 --temperature 0 --temperature 0.2 --max-tokens 256

awb patch-template

Creates editable implementation templates:

awb patch-template --proposal-id prop-1

awb promote

Promotes a candidate to baseline:

awb promote --experiment candidate-id
awb promote --experiment candidate-id --baseline baseline-id

awb ablate

Tests surface ablation:

awb ablate --agent-id my-agent --eval-id my-eval --surfaces tools
awb ablate --agent-id my-agent --eval-id my-eval --surfaces prompts,tools --limit 4

awb matrix

Cross-agent/provider matrix:

awb matrix --agent-id agent-a --agent-id agent-b --eval-id my-eval

awb factorial

Factorial experiments:

awb factorial --experiment exp-id

---

Scoring

Default Scoring

The default scorer bundle includes:

• output_quality (0.7): Did the output match expectations?
• process_alignment (0.3): Did the agent use expected tools/skills?

Scorer Bundles

Predefined scorer bundles:

Bundle	Use Case
default	General purpose
support_agent	Support desk agents
coding_agent	Code generation agents
retrieval_agent	RAG/retrieval agents
router_agent	Decision/routing agents
release_gate	Release decision agents

Custom Scoring

Add score_cases() to .agent-workbench/scorers/<eval-id>_default.py:

from agentworkbench.scorer_utils import score_output_quality, score_process_alignment
from agentworkbench.scorers.workflow import score_conversation_turns

def score_cases(run_artifact: dict, case: dict) -> list[dict]:
    scores = [
        score_output_quality(run_artifact, case),
        score_process_alignment(run_artifact, case),
    ]
    turn_scores = score_conversation_turns(run_artifact, case)
    scores.extend(turn_scores)
    return scores

Turn-Level Scoring

For multi-turn cases, define expected_turns in case metadata:

{
  "metadata": {
    "expected_turns": [
      {"step": 0, "expected_tools": ["policy_search"], "expected_skills": ["policy_grounding"]},
      {"step": 1, "expected_tools": ["customer_memory"], "expected_skills": ["memory_recall"]}
    ]
  }
}

This generates per-turn scores: turn_0_alignment, turn_1_alignment, etc.

---

Configuration

Workbench Config

.agent-workbench/workbench.yaml:

artifacts_dir: .agent-workbench/runs
snapshot_mode: auto
env_files:
  - .env.providers.local
gates:
  min_average_score: 0.75
  min_passed_case_delta: 0
  require_clean_benchmark: true
  max_regressed_cases: 0
loop_checks:
  enabled: true
  fail_on_severity: critical
  block_promotion_on_critical: true
  large_file_line_threshold: 500
  large_function_line_threshold: 80
  min_rationale_chars: 12
  expected_string_match_min_chars: 8

Acceptance Gates

Gate	Description
min_average_score	Minimum average score to pass
min_passed_case_delta	Minimum improvement in passed cases
require_clean_benchmark	No benchmark changes allowed
max_regressed_cases	Maximum allowed regressions
min_stability_runs	Minimum runs for stability check

---

Architecture

Models

Key data models in agentworkbench/models.py:

• AgentSpec: Agent definition
• BehaviorBrief: What "better" means
• EvalSuiteSpec: Eval suite configuration
• EvalCase: Individual test case
• RunArtifact: Execution result
• ExperimentRecord: Complete experiment
• ComparisonRecord: Baseline vs candidate comparison
• ChainStepConfig: Chain step configuration
• ConversationMetrics: Multi-turn conversation stats

Adapters

Adapter	Description
command	Run shell commands
python	Call Python functions
provider	OpenAI-compatible API calls

Adapter Environment Variables

• AWB_INPUT — Case input
• AWB_TURNS_JSON — Multi-turn conversation
• AWB_CHAIN_CONTEXT_JSON — Chain context
• AWB_PROMPTS_JSON — Prompt definitions
• AWB_TOOLS_JSON — Tool definitions
• AWB_CASE_PATH — Case file path
• AWB_OUTPUT_PATH — Output file path

Scorers

Located in agentworkbench/scorers/:

• output_quality — Output matching expectations
• process_alignment — Tool/skill usage
• grounding — Evidence-based answers
• json_schema — Structured output validation
• safety — Forbidden term checking
• workflow — Turn-level alignment

---

Examples

See examples/ for complete examples:

• live-agent-demo/ — Local command adapter demo
• live-provider-demo/ — Provider (Groq/Cerebras) demo

Running Examples

cd examples/live-provider-demo

# Run smoke tests
awb run --agent-id groq-support-baseline --eval-id support-ops-agent --subset smoke

# Run multiturn tests
awb run --agent-id groq-support-baseline --eval-id support-ops-agent --subset multiturn

# Run chain tests
awb run chain \
  --agent-id groq-support-baseline \
  --agent-id groq-router-workflow \
  --eval-id support-ops-agent \
  --subset chain

# Run routed workflow tests
awb run workflow \
  --workflow-file groq-routing-workflow \
  --eval-id workflow-routing-live \
  --subset full

# Generate and run a red-team smoke suite
awb redteam generate \
  --source-eval-id provider-tool-interface \
  --output-eval-id provider-tool-interface-redteam \
  --mode adversarial \
  --mode robustness \
  --mode stress
awb evals approve --eval-id provider-tool-interface-redteam
awb run --agent-id groq-provider-tool-interface --eval-id provider-tool-interface-redteam --subset smoke

# Diagnose
awb diagnose --eval-id support-ops-agent --subset chain

---

Output Interpretation

Experiment Results

{
  "experiment": {
    "experiment_id": "...",
    "agent_id": "groq-support-baseline",
    "eval_id": "support-ops-agent",
    "average_score": 0.65,
    "passed_cases": 3,
    "total_cases": 5
  },
  "aggregate": {
    "metric_averages": {
      "output_quality": 0.6,
      "process_alignment": 0.7,
      "turn_0_alignment": 1.0
    },
    "behavior_summary": {
      "average_turn_count": 2.0,
      "escalation_rate": 0.0,
      "tools_used": ["policy_search", "incident_lookup"]
    }
  }
}

Chain Diagnosis

{
  "chain_diagnosis": {
    "is_chain_run": true,
    "weakest_step": 0,
    "weakest_agent": "groq-support-baseline",
    "step_failure_counts": {
      "0": {"groq-support-baseline": 2},
      "1": {"groq-router-workflow": 1}
    }
  }
}

---

Troubleshooting

awb doctor

Run diagnostics:

awb doctor

Checks:

• Required environment variables
• Repo structure
• Config validity

Common Issues

Missing API Keys

Set required environment variables:

export GROQ_API_KEY=your-key
export CEREBRAS_API_KEY=your-key

Scorer Not Found

Ensure scorer file exists at .agent-workbench/scorers/<eval-id>_default.py

Adapter Errors

Check agent spec adapter configuration and entrypoint validity.

---

Security

Please report vulnerabilities via the process in SECURITY.md.

Code of Conduct

This project follows the Contributor Covenant. See CODE_OF_CONDUCT.md.

License

MIT — see LICENSE.