verdictkit

Non-custodial verification for agent work. Answers one question: did the agent do what it said it would?

The product is the verification layer, not custody: a machine-readable contract (intended action + checkable success criteria), deterministic checks against the world, and a rubric verdict — PASS / FLAG / FAIL — with evidence. No money moves through it, so there is no money-transmitter licensing wall. That is the whole strategic point.

Status: 0.1.0, launch-ready, NOT published to PyPI. Do not run the publish command without approval.

Quickstart

pip install verdictkit        # not on PyPI yet; use: pip install .

Local verification (free, unlimited, no key, no network)

import verdictkit as vk

contract = vk.define_contract(
    task_id="research-042",
    agent_id="agent-7f3a",
    intended_action="Summarize the Q3 incident reports into incidents.md",
    success_criteria=[
        {"id": "c1", "type": "file_exists",
         "path": "incidents.md",
         "description": "summary file was written"},
        {"id": "c2", "type": "pattern_count_gte",
         "path": "incidents.md",
         "pattern": r"^## Incident",
         "min_count": 3,
         "description": "covers at least 3 incidents"},
    ],
    unverifiable_claims=["tone is executive-appropriate"],  # -> FLAG, honestly
)

result = vk.verify_run(contract, base_path="./output")
print(result["verdict"])   # PASS | FLAG | FAIL
print(result["reason"])    # evidence, always

Or from the shell:

verdictkit verify contract.json --base-path ./output -v
# exit code: 0 = PASS, 1 = FAIL, 2 = FLAG, 3 = usage/IO error

Hosted verification (reputation ledger + re-verification)

Local mode checks your machine. The hosted tier lets a third party trust the verdict: you capture evidence locally, the server re-runs the deterministic checks against the evidence — it never trusts the agent's claims alone.

import verdictkit as vk

client = vk.Client(api_key="vk_live_...")          # base_url=... to override
contract_id = client.create_contract(contract)      # -> "vc_..."
bundle = vk.collect_evidence(contract, base_path="./output")
verdict = client.submit_run(contract_id, bundle)    # server-side verdict dict

client.get_verdict(contract_id)                     # latest verdict
client.list_verdicts(agent_id="agent-7f3a")         # reputation ledger

Errors are clean: vk.AuthError on bad credentials (401/403), vk.VerdictkitError on network trouble, timeouts, or bad responses.

API reference

Verdict shape

{"contract_id": ..., "task_id": ..., "agent_id": ...,
 "verdict": "PASS" | "FLAG" | "FAIL",
 "reason": "human-readable evidence summary",
 "check_results": [{"check_id": ..., "type": ..., "status": "pass"|"fail"|"error",
                    "evidence": ...}, ...],
 "unverifiable_claims": [...]}

Check types

file_exists · file_nonempty · file_contains (regex) · pattern_count_gte · dir_exists · json_valid · line_count_gte · sqlite_table_has_rows · path_absent (scope guard)

The rubric

• FAIL — any check fails. Fail dominates everything else.
• FLAG — no failures, but something couldn't be verified: a check errored, a claim was declared unverifiable, or there was nothing checkable at all.
• PASS — every check passed cleanly, and nothing was left unverifiable.

Design rule: a check that cannot run is an error, never a pass. A run with nothing checkable can at best be flagged. Absence of evidence is never evidence of success.

Trust model

1. Contracts are machine-readable promises. What the agent intended to do, written before the run, with deterministic success criteria.
2. Evidence is content-addressed. collect_evidence captures file bytes with their sha256, directory listings, and query results. The bundle carries a bundle_hash over canonical JSON — the server recomputes every hash; tampering breaks them.
3. The server re-runs checks; it never trusts claims. Hosted verification executes the same deterministic check engine against the bundle's content. Claims without evidence stay in unverifiable_claims and can only produce FLAG.
4. Truncation is honest. Artifacts are capped (256 KiB file content, 2000 dir entries). Truncated artifacts are marked, and checks the server cannot fully re-run become FLAG — never PASS.

What verification is NOT

• Not a guarantee of quality. PASS means the deterministic criteria were met — the file exists, has ≥3 sections, parses as JSON. It says nothing about whether the writing is good, the code is correct, or the summary is accurate. Put anything subjective in unverifiable_claims and accept the FLAG honestly.
• Not an oracle. Checks only see what they can read: files, dirs, sqlite. They cannot verify "the user is happy" or "the bug is fixed" unless that was operationalized into a checkable artifact.
• Not custody, not escrow. No money moves through verdictkit; it attests to work done, it does not hold funds or release payment.
• Not tamper-proof on a compromised machine. Local verification trusts the local filesystem. The hosted tier raises the bar (content-addressed bundles, server-side re-runs, reputation ledger) but a fully compromised agent host can still fake the inputs. Verification makes lying auditable, not impossible.

Development

pip install -e ".[dev]"   # dev extras: pytest, build
python3 -m pytest         # 59 tests, all local, no network
python3 -m build          # wheel/sdist -- DO NOT upload without approval

License

MIT. See LICENSE.