skill-harness

skill-harness is a measurement instrument for controlled skill-vs-no-skill comparisons. It measures what a skill changes, what the skill costs, and whether the evidence is strong enough to justify a claim.

It is for a person who writes or maintains agent skills and has to decide whether to ship one.

Install

pip install skill-harness                              # Python 3.12 or later
skill-harness skill audit path/to/your/SKILL.md

Section 5 below explains what the audit reports and how to run a paid comparison.

One line carries the design:

skill exists ≠ skill was delivered ≠ model used skill

A skill exists when the file is present, which says almost nothing about what the model received. It was delivered when the runtime put its description into the model's context, which this instrument calls exposure and treats as the treatment condition. The model used it when it loaded the body, which is invocation, a downstream behavior the instrument records rather than the treatment. In a 32-epoch paired run on git-pull-rebase-trap (an epoch is one attempt at the task), exposure held at 32 of 32 while the body was invoked in 24 of 32 (receipt). Reading delivery as use would have been wrong about a quarter of the treated arm.

Why this exists

I wanted to know if you could tell if a skill was any good.

That sentence is the owner's. The long account: docs/why-this-exists.md.

1. What problem does this solve?

A skill costs context in every conversation, whether or not it fires, and that cost is arithmetic on text. The benefit is conditional and hard to establish.

Suppose you run a task with a skill and score 85, and without it you score 75. The naive conclusion is that the skill improved performance by 10 points. You have not established that. Maybe the skill was never delivered, or was delivered differently between runs. Maybe the environment changed, or the model changed, or the evaluator behaved differently, or one run was contaminated. Put the new tire on one car and the old tire on another and you learn nothing about the tire.

The ratified question splits that in two:

What does this skill cost you, and which parts of it are worth that cost?

2. What does the instrument do?

It constructs controlled comparisons where the treatment can be independently verified, records runtime and evidence provenance, refuses inadmissible results, and reports only what the experiment establishes.

Cost and benefit never collapse into one score. Cost is mechanical, measurable with no model spend. Benefit needs a controlled comparison, and the question there is not whether the subject can do the task but whether the skill changes performance against an otherwise comparable condition. That separation is why a skill's benefit can be UNMEASURED without the skill being useless.

Four layers do the work: subject execution under a controlled Full condition (the complete skill) or Null condition (no skill), evidence normalization of runtime-native records into one representation that keeps their provenance, a scientific decision layer holding the registered treatment definition, the evidence-admissibility rules, the oracle and the statistics, and reporting of what survived them. They are separate so that the runtime does not get to define what counts as scientific evidence.

A screen returns one of three verdicts: KEEP, CUT with a named reason (subsumed, meaning the model was already doing it; no_lift; or harmful), or CANT_TELL_YET. Which verdict a skill is eligible for depends on its registered value class, not the numbers alone, and the value-class guard enforces that: subsumed is a CUT only for skills registered TRANSFORMATIVE_LIFT, the class whose whole claim is lift above the bar. append-only-evidence-design and a hardened git-pull-rebase-trap both returned CUT (subsumed) under the pre-guard rule, each at a no-skill pass rate of 1.00; the guard reclassified both to CANT_TELL_YET, and the pre-guard results stay in the record as dated output.

3. What does it refuse to claim?

The governing rule: where no measurement exists, record the absence of measurement rather than inventing a number or a verdict. UNMEASURED is never bare. Which kind of not-knowing applies is more information than not-knowing alone, so it always carries a sub-reason from a closed vocabulary. That vocabulary is fixed in one place, the UnmeasuredSubReason enum in src/skill_harness/aggregation/status.py, and every member is defined in docs/concepts/why-unmeasured.md.

The oracle only sees the final filesystem. The registered oracle runs against the isolated final filesystem, so it measures task outcome only. It cannot observe program design, coupling, scope creep, or test quality. A skill can therefore raise the oracle result while degrading the codebase, and the instrument will correctly report KEEP. Maintainability effect is UNMEASURED for every skill result produced to date. No existing claim is retracted. The scope is now stated.

The runtime is part of the apparatus. Cross-runtime pooling is not automatic and is not permitted without a separate equivalence measurement. The Pi subject adapter is implemented, and its refusal predicates and its driver wiring carry mutation receipts over seven mutants, all killed, one of which shows the pre-spend identity gate refusing an undeclared evidence pipeline before any spend (refusal, driver wiring). The adapter measures the runtime version from the binary that ran and pins no version itself. No sized Pi measurement has run, so no Pi skill effect is admissible yet. Apparatus compatibility is established and nothing more.

Zero production-skill KEEPs. The one KEEP on record is a declared synthetic positive control, 8 of 8 with the skill against 0 of 8 without: the instrument fires when an effect is present, and that says nothing about any real skill. A sized benefit run launches only on the first task whose no-skill screen returns a pass rate below 1, and the production screens ceiling at 1. Every verdict, with its receipt: what the instrument has found.

The rule points inward; both figures stay here.

Extraction repeat-variance: MEASURED for one skill. Three repeat extractions of the same SKILL.md returned 29/33/34 clauses, so clause counts are not stable run to run, and nothing downstream keys on clause position (#152).

Vacuity-flag precision: MEASURED at 0.972 by blind cross-family adjudication over 106 adjudicated rows, flag-level only (#153, receipt docs/calibration/vacuity-flag-calibration-2026-08-08.json). When the adjudicators also had to agree on which kind of vacuity, kind-precision 0.835: not_a_directive matched 77/77, while weak_directive matched 4/20. The vacuity-flag detector's recall is UNMEASURED: the unflagged clauses were never adjudicated.

4. How does it keep the comparison honest?

Six properties are explicit. Each is a way a comparison goes wrong when left implicit.

• Treatment delivery. The ingest path checks exposure per epoch on both arms before it writes evidence. A Full epoch with no exposure raises UnexposedFullEpochError, a Null epoch with exposure raises NullArmContaminationError (src/skill_harness/subject/ingest.py), and both are recorded as apparatus errors, not evidence.
• Subject and runtime identity. Every receipt pins the subject, the harness version, the metric version and an implementation hash.
• Evidence admissibility. Evidence passes a gate before it enters an aggregate, and the append-only store snapshots the gate result at write time. It keeps what fails the gate and never counts it, so an auditor can check the gate.
• Oracle identity. The outcome rule is registered and versioned with the result, and a judge-graded result counts only where that judge was calibrated on that axis first.
• Ratification. A paid run binds to a decision record before it spends.
• Provenance. Every figure carries instrument identity, the model pin and prompt fingerprint that stamp which generation produced it. Models change underneath a figure, so two numbers from two generations are visibly non-comparable rather than averaged.

The Skill Efficacy Reporting Standard (SERS) fixes that vocabulary, as a JSON Schema plus a prose companion in docs/sers/. CI checks that this repository's own receipts validate against it, that its enums match the code's, and that deliberately poisoned receipts are rejected.

5. How do I use it?

Start with the zero-cost offline audit. Move to a ratified paid comparison only when the evidence path justifies it. The install and audit commands are at the top of this page.

The audit makes no API call, needs no key, and touches no network. It reports the cost triple (standing, fired, aux) as arithmetic on text, checks structure against Anthropic's authoring spec, and states what a paid run could measure today.

Read the gaps it reports. Run on 2026-09-13 from a clean v0.3.0 install against the fixture tests/fixtures/sers/declared-synthetic-positive-control/SKILL.md, the audit printed Standing cost (mechanical): UNMEASURED and the check name standing-cost-unparseable, because its minimal parser could not read that card's description. A default there would have understated the per-turn tax. That whole run is in docs/offline-audit-example.md. --strict exits 1 on warnings, for CI.

A paid comparison:

skill-harness skill init path/to/SKILL.md --execute   # extract testable claims
skill-harness run ablation <skill_id>                 # dry-run: no calls, no cost
skill-harness run ablation <skill_id> --execute \
  --ratification docs/ratifications/RAT-NNNN-<skill-slug>.md \
  --task-family <family> --estimand <treatment-policy|hypothetical>
skill-harness run evaluate-skill <skill_id>           # aggregate to a verdict

Set ANTHROPIC_API_KEY or OPENROUTER_API_KEY. Spending is a per-subcommand property.

run ablation and run pi-paired are the subcommands that spend. run ablation and run evaluate-skill read those same two database paths; pass --evidence-db or --runtime-db to put them elsewhere. The three flags on run ablation --execute must match the scope fields of a record in docs/ratifications/ (#47), and that preflight runs before any model call and before the API-key check, so a run without one exits 1:

Error: ratification preflight refused --execute [record-missing]: no ratification record referenced; pass --ratification docs/ratifications/RAT-NNNN-<skill-slug>.md (see that dir's README)

A skill you did not write

A skill under measurement is untrusted input. run pi-paired loads the skill files into the Pi coding agent, and that agent holds tools. Pi's default tools are read, bash, edit and write. The run config's tool_allowlist replaces that set, so it can remove tools or add others. A hostile skill can steer whatever tools the agent holds. run ablation sends text-only model requests with no tools, and run evaluate-skill makes no model call.

When the run config has no container object, the Pi epoch and the oracle run on your host, as your user, with your filesystem and your network. The pin records this as local-host:no-container. Measure a skill you did not write with a container object:

{
  "container": {
    "argv_prefix": [
      "docker", "run", "--rm",
      "-e", "ANTHROPIC_API_KEY", "-e", "PI_CAPTURE_DIR",
      "-v", "/abs/run-root:/abs/run-root",
      "-w", "/abs/run-root/workspace",
      "registry.example/pi-runner@sha256:<digest>"
    ],
    "image_digest": "sha256:<digest>",
    "network_policy": "egress to the model provider only"
  }
}

This is a template, not a tested recipe. No run in this repository has used container mode. Four facts bound it:

• The harness puts argv_prefix in front of the Pi command and the oracle command, and does nothing else with the container. The isolation is whatever that prefix builds. The harness records image_digest and network_policy in the pin and enforces neither (#812).
• The launcher hands the container host paths. The skill directories, the session directory and the capture extension inside the installed package go on the command line. The workspace is the working directory, which the container does not inherit, so set it with -w. The capture writes into the artifact directory that PI_CAPTURE_DIR names, and the host reads it back after the epoch. Mount each of these at the same path inside the container. A missing artifact mount loses the capture, and the harness refuses the pair after the epoch has already spent.
• The epoch's environment holds only declared_env, the secret_env_names values and PI_CAPTURE_DIR. Pass each name the container needs with -e.
• The agent calls the model provider from inside the container. Docker's --network none removes that route, so the epoch cannot run. Restrict egress to the provider instead.

The other half

The verdicts land in MrBinnacle/skills. There, each skill carries its own dated evidence record, and controlled results are read from that skill's record, not a front-page roll-up. Skills are re-screened when a major model ships and publicly retired, with the record intact, once the model no longer needs them or a platform change meets a pre-registered trigger. Each retirement is made against its stated criterion.

Where the rest is

• What the instrument has found: every verdict to date, with its receipt.
• Run-to-run noise: why a three-runs-a-side comparison cannot settle this.
• Other tools in this area, and what this repository does not claim.
• Reading guide: the case studies, the receipts index and docs/PRD.md.

Status: v0.4.0 is the latest published release, on PyPI. The published package and the current development tree are distinct states. Not every older screen record is in the evidence store yet. Work is prioritised correctness first, then evidence integrity, runtime portability, measurement breadth and presentation.

MIT licensed. Issues and PRs welcome: CONTRIBUTING.md.