agent-tune-kit 0.4.3

Local Codex plugin for iterative Agent tuning with guided Skills, reusable runner templates, versioned results, and static validation.

Agent Tune Kit

English | 简体中文

Agent Tune Kit is a local Codex plugin for evaluating and tuning your own local Agent.

If you already have a working Agent but do not know where it fails, why it fails, or what to change next, Agent Tune Kit helps you run the full loop: batch test the Agent, find failure cases, generate a report, let Codex tune the Agent, and verify the next run.

Architecture

Who It Is For

Use it if you have, or want Codex to help you fill in:

• A local Agent, chatbot, tool-using Agent, or RAG Agent.
• A small evaluation dataset, preferably CSV; 5 to 20 rows are enough to start.
• Inputs, expected answers, or human-checkable results.
• A desire to let Codex help locate weak spots and tune prompts, code, parameters, or tool configuration.

Install

One-command install:

uvx --from agent-tune-kit atk install

To keep the atk command available:

uv tool install agent-tune-kit
atk install

Or use pipx:

pipx install agent-tune-kit
atk install

After installation, open the plugin list in Codex:

/plugins

Select and enable Agent Tune Kit. If $atk-status and other completions do not appear immediately after enabling, restart Codex or reopen the current project session.

Minimal Tuning Loop

Run these commands in your Agent project, not in this repository.

Ideally, you already have a local Agent project that Codex can inspect and edit, plus an evaluation dataset. CSV is recommended, but column names do not need to follow a strict schema; Codex will infer inputs, expected results, and evaluation shape from the data. If either piece is missing, start with step 0. If both already exist, go straight to step 1.

0. Optional: Fill In the Dataset or Agent

If you only have a business description, examples, or acceptance rules, run:

$atk-build-dataset <your business description, examples, or rules>

Codex asks 1-3 questions when information is insufficient, prioritizing input fields, expected output or acceptance criteria, and key business scenarios. The result is written directly to .atk/datasets/dataset.csv with atk_id; if that file already exists, Codex asks before overwriting it. The dataset focuses on main flow, boundary input, missing or ambiguous information, refusal/uncertainty, output format constraints, and business risks you describe.

If you already have an evaluation dataset but do not have an Agent project yet, generate a small runnable Python Agent that uses an OpenAI-compatible API:

$atk-new-agent dataset is data/eval.csv

Codex inspects the dataset, clarifies your intent, generates a minimal Agent project, and writes the interview and design notes to .atk/specs/agent_spec.md. This step does not write .atk/datasets/dataset.csv; $atk-init still owns dataset validation, normalization, and runner generation when connecting the Agent.

If .atk/datasets/dataset.csv already exists but the expected-result semantics are missing or too weak for stable failure judgment, enrich the existing dataset before running evaluation:

$atk-build-ground-truth

Codex asks you to choose one dataset-wide ground_truth style: exact answer or natural-language acceptance criteria. It then updates .atk/datasets/dataset.csv in place, preserving existing columns and atk_id. This is a dataset-only, pre-results step: it does not run the Agent, create .atk/results/vN, or write failure_cases.csv. If existing eval_results.csv was produced before this enrichment, rerun $atk-run before finding failures. Once the dataset is built, you can optionally review and quality-check it first:

$atk-visualize-dataset

It renders .atk/datasets/dataset.csv into a local, single-file, offline HTML browser:

.atk/datasets/dataset.html

The page supports search, category filtering, and paginated row browsing, highlights the input-vs-ground_truth comparison, and auto-flags issues such as empty values, duplicate/missing atk_id, conflicting samples, and length outliers, so you can quickly confirm whether each ground_truth matches expectations. You can mark a verdict per row (matches / suspect / needs fix) and export dataset_review.csv, then return to $atk-build-dataset to fix it.

1. Initialize

Tell Codex where your Agent starts and where the evaluation data lives:

$atk-init My Agent entrypoint is scripts/agent.py and the evaluation dataset is data/eval.csv

Codex generates:

.atk/runner/eval_runner.py

If the Agent was created by ATK new Agent, the next command is usually:

$atk-init Agent entrypoint is agent.py run_agent and the evaluation dataset is data/eval.csv

2. Run Evaluation

$atk-run

Results are written to:

.atk/results/v1/eval_results.csv

3. Find Failures

Let Codex judge which rows failed:

$atk-find-failures

If you already have a clear rule, create the rule script first and then apply it:

$atk-init-failure-rule rule: mark a row as failed when expected differs from agent_output
$atk-find-failures-by-rule

Failure cases are written to:

.atk/results/v1/failure_cases.csv

4. Generate Report

$atk-report

The report is written to:

.atk/results/v1/report.md

It summarizes results, failure cases, likely causes, and recommended tuning priorities.

5. Optional: Browse Failures

$atk-visualize-failures

This creates a local HTML page:

.atk/results/v1/failure_cases.html

Use it to search, filter, and manually review failure cases.

6. Let Codex Tune the Agent

$atk-tune

Codex edits your Agent based on the report and records the tuning plan:

.atk/results/v1/tuning_plan.md

Verify Improvement

After tuning, run another loop:

$atk-run --only-failures
$atk-find-failures
$atk-report

New results are written to .atk/results/v2/. --only-failures maps the prior failure_cases.csv back to .atk/datasets/dataset.csv by atk_id and reruns only those rows. Starting with the second loop, the report compares against the previous tuning_plan.md and tells you whether the target issues were resolved, partially resolved, unresolved, or impossible to judge.

Output Structure

.atk/
├── datasets/
│   └── dataset.csv        # ATK runnable dataset with atk_id
├── runner/
│   ├── eval_runner.py
│   └── failure_rule.py
└── results/
    ├── v1/
    │   ├── eval_results.csv
    │   ├── failure_cases.csv
    │   ├── failure_cases.html
    │   ├── report.md
    │   └── tuning_plan.md
    └── v2/
        └── ...

Common output files:

• eval_results.csv: actual Agent output for each row.
• failure_cases.csv: rows selected as failures.
• failure_cases.html: optional failure review page.
• report.md: analysis and tuning recommendations.
• tuning_plan.md: what Codex changed and why.

Common Skills

• $atk-status: inspect progress and suggest the next step.
• $atk-build-dataset: build .atk/datasets/dataset.csv from business context, examples, or rules.
• $atk-build-ground-truth: enrich an existing .atk/datasets/dataset.csv with a canonical ground_truth column.
• $atk-visualize-dataset: render .atk/datasets/dataset.csv into a local HTML browser for quickly reviewing rows and confirming whether each ground_truth matches expectations.
• $atk-new-agent: create a lightweight OpenAI-compatible Agent when you only have a dataset.
• $atk-init: generate the test runner.
• $atk-run: run evaluation and create a new result version.
• $atk-find-failures: let Codex identify failure cases.
• $atk-init-failure-rule: create or update the failure rule.
• $atk-find-failures-by-rule: apply the rule to identify failures.
• $atk-report: generate analysis and cross-loop validation.
• $atk-visualize-failures: generate the failure review HTML page.
• $atk-tune: tune the Agent based on the report.