VERDICT

Evidence-governed admission of self-evolved skills for small LLM agents.

Self-improving agents keep an edited skill when it scores higher than its parent. With small models, repeated runs of an unchanged skill already differ by several points, so many of those "improvements" are noise: in our measurements keep-if-better admits a large share of skills that did not change at all (noise floors and gate simulations in results/tables/noisegate/). VERDICT is the package that decides, from paired evidence, whether an evolved skill is really better, and records each decision as an admission card.

Status: pre-release. The verdict package (paired evidence, policies, evidence cards, replay, budget guard, freeze manifests; CLI verdict) is working and tested; the distribution name is not final (see "Release status").

Repository layout

src/verdict/      the VERDICT package: core/ (statistics, policies, cards), adapters/, runners/, budget/, cli.py
tests/            offline tests for verdict (synthetic fixtures in tests/fixtures/)
schemas/          evidence_card.schema.json, the JSON Schema every card is validated against
examples/         long-format CSV import format and an offline quickstart
configs/          policies, frozen splits, experiment configs, noise floors (frozen at prereg-v1)
prereg/           pre-registration and freeze manifests (checked by scripts/freeze_check.py)
scripts/          experiment runners (paid, budget-guarded) and offline analyses
harness/          experiment harness that produced the results (own pyproject, see harness/README.md)
  metaskill/      from-scratch replication of MetaSkill-Evolve (arXiv:2607.05297): SealQA, OfficeQA, ALFWorld, ALE, GSM8K
  noisegate/      NoiseGate, the stdlib-only admission gate prototype (noise floor, paired tests, admission card)
  scripts/        run wrappers and analyses (index in harness/scripts/README.md)
  configs/        one YAML per model configuration; gate settings; ALE presets
  data/           frozen splits (SHA256SUMS) and a 10-question OfficeQA sample
  tests/          offline tests for metaskill and the scripts
results/          every result, as small files: run summaries, per-item outcomes, tables, ledger (see results/README.md)
docs/             replication notes, research records (plans, decisions, run log), release checklist (see docs/README.md)

Install

uv sync                  # verdict + its dev group (pytest, ruff, the harness workspace member the tests need)
cp .env.example .env     # only for commands that call a model or a gated dataset; nothing below needs it

From PyPI: pip install verdict-gate (the import name and the CLI stay verdict).

Requires Python 3.12+ (uv recommended; .python-version pins 3.13). Without uv, pip install . (or pip install -e .) in a checkout installs the verdict package alone (numpy, pydantic, pyyaml, jsonschema, typer), without the harness. A wheel carries the card schema and the policy, validity, strata and floor configs it needs (verdict/_data); inside a checkout the package reads schemas/ and configs/ from the checkout.

Quickstart (offline)

uv run python examples/quickstart.py

It imports examples/long_format.csv (the generic import format, examples/README.md), then decides three synthetic candidates against their parent under keep-if-higher and under the moderate policy, and writes one evidence card (card.json, valid against schemas/evidence_card.schema.json, and card.md). The same decision from the CLI:

uv run verdict compare --baseline tests/fixtures/replay/s0.json --candidate tests/fixtures/replay/c_good.json --policy configs/policies/moderate.yaml --fnoise tests/fixtures/replay/fnoise_m.yaml --out card.json
uv run verdict card render card.json

verdict compare exits with a code per decision (0 = PASS_GLOBAL; see src/verdict/cli.py), verdict replay recomputes decisions for many arms, verdict freeze writes a freeze manifest, verdict budget status reads the spend ledger.

Tests

uv run pytest            # all offline tests: no network (blocked by conftest.py), no API keys, no raw runs needed
uv run ruff check .

Reproducing the results

None of the commands below makes a model call or needs a key; they read tracked files only.

Every number in results/ cites the run it came from; docs/plan/RUNLOG.md (thesis experiments) and docs/research/RUNLOG.md (earlier NoiseGate work) give the exact command of each logged run. The paid runners (scripts/run_*.py) all have a --dry-run that prints the planned runs and the estimated cost; model calls go through OpenRouter and cost money.

Not yet reproducible from tracked data: the E3 decision analysis (scripts/analyze_e3.py), H48 Event 1, the E8 evolution and H48 Event 2 read raw arm files under runs/ (not tracked). Harness commands run from harness/; see harness/README.md.

Data and licenses

Code is licensed under Apache-2.0. Datasets keep their own licenses. What is in this repository:

All of the above can be redistributed in this form; the CC-BY-4.0 and CC-BY-SA-4.0 rows need the attribution kept.

Release status

The distribution name is still open (RUNLOG D10: verdict is taken on PyPI); it lives only in pyproject.toml and src/verdict/, so the rename is one change. The implementation spec's revision-eval CLI is the verdict command here.

Citation

A CITATION.cff will be added with the thesis title. Until then, cite MetaSkill-Evolve (arXiv:2607.05297) for the method the harness replicates.