pisama-verifier-gym

Audit harnesses for verifiers: LLM judges, reward functions, graders, and failure detectors. The package ships a verifier datasheet template, a worked WildChat derailment judge datasheet, sanitized backing artifacts, and a no-dependency agreement calculator.

Install

pip install pisama-verifier-gym

Quick Start

pisama-verifier-gym agreement
pisama-verifier-gym validate artifact.json
pisama-verifier-gym gate baseline.json candidate.json
pisama-verifier-gym render artifact.json --output datasheet.md

from pisama_verifier_gym import agreement_table, load_builtin_verdicts

rows = load_builtin_verdicts()
for stat in agreement_table(rows):
    print(stat.vendor_a, stat.vendor_b, stat.raw_agreement, stat.positive_specific_agreement)

What Is Included

• TEMPLATE.md: the verifier datasheet template.
• datasheets/derailment-wildchat.md: a filled worked example for a task derailment LLM judge.
• data/wildchat_v3_derailment_verdicts.jsonl: sanitized per-trace panel verdicts, joinable to WildChat by source_trace_id.
• data/contested_adjudication.sanitized.json: contested-label adjudication record with lineage fields and conversation text removed.
• data/judge_agreement.json: aggregate agreement artifact from the same lane.

Conversation text is not redistributed. WildChat is distributed by AI2 under its own license terms.

Python API

from pathlib import Path
from pisama_verifier_gym import (
    agreement_table,
    load_verdict_rows,
    pairwise_agreement,
    verdict_balance,
)

rows = load_verdict_rows(Path("verdicts.jsonl"))
table = agreement_table(rows)
balance = verdict_balance(rows)

anthropic_google = pairwise_agreement(
    rows,
    "claude-sonnet-4-6",
    "gemini-2.5-flash-lite",
)

Each pair reports:

• usable row count after pairwise abstention drops
• raw agreement
• positive specific agreement
• Cohen's kappa

Read raw packaged assets:

from pisama_verifier_gym import read_datasheet, read_template

print(read_template())
print(read_datasheet("derailment-wildchat"))

CLI

# Built-in WildChat derailment verdicts
pisama-verifier-gym agreement

# A custom JSONL export with the same per_vendor_verdicts shape
pisama-verifier-gym agreement path/to/verdicts.jsonl

# Machine-readable output
pisama-verifier-gym agreement --json

# Validate the Verifier Gym contract
pisama-verifier-gym validate artifact.json

# Compare two artifacts with the same verifier ids
pisama-verifier-gym compare baseline.json candidate.json --json

# Fail on F1 drops, PSA collapse, abstention spikes, threshold drift, or
# unexpected fingerprint changes
pisama-verifier-gym gate baseline.json candidate.json

# Render machine-generated datasheet tables as Markdown
pisama-verifier-gym render artifact.json --output datasheet.md

# Export Pisama calibration reports into the gym contract
pisama-verifier-gym export-calibration backend/data/calibration_report.json \
  --llm-report backend/data/llm_detector_calibration.json \
  --output backend/data/verifier_gym/current.json \
  --positive-manifest backend/data/verifier_gym/positive_rich_manifest.json

The validator fails hard when a verifier lacks rubric lineage, a dataset fingerprint, an input visibility policy, a lane policy, or when synthetic data can feed published metrics. The gate is intentionally fingerprint-aware: by default a candidate must be compared against the previous run for the same dataset fingerprint.

Why This Exists

High raw agreement is not enough when the positive class is rare. The included WildChat example shows raw agreement of 0.96 to 0.98, while positive specific agreement is 0.00 on the same slice. That distinction decides whether a published verifier metric is useful or misleading.

Development

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
ruff check src tests
mypy src/pisama_verifier_gym
pytest -q
python -m build

Internal Pisama development keeps checked-in gym artifacts under backend/data/verifier_gym/. Regenerate them from real calibration outputs with python backend/scripts/verifier_gym/export_current.py, then validate, render, and gate before promoting a new baseline.

License

Code, documentation, and sanitized artifacts in this package are MIT licensed.