Witnessbox

Evidence-based testing (EBT) of application behaviour: during tests the code records which decision it made and on what data, and an independent checker re-verifies every decision against a human-approved specification. The kit consists of a run gate, a checker and SDKs for Java, Dart and TypeScript. The runtime requires Python 3.11+ and has no third-party dependencies. The gate is a bash script tested on macOS and Linux; native Windows is not supported, and running it under WSL is not covered by tests.

Quick start

uv tool install witnessbox          # or: pipx install witnessbox
witnessbox scaffold ts ebt          # ebt/ scaffold for the dart, ts, jvm or generic stack
witnessbox sync-pinned ebt          # copy the SDK pinned in ebt/pinned.txt into the project
# write an evidence test and ebt/prepare.sh, then commit ebt/ and the project
witnessbox test-checks ebt          # unit tests of the project checker
witnessbox gate --check-only ebt .  # wiring check; runs no tests
BASE=$(git rev-parse HEAD) witnessbox gate ebt . ../results   # gate: PASS, exit code 0

The scaffold contains an example spec.toml, checks.py and a stack-specific test.sh. A human approves the specification before the first run, and test.sh wires in the project's tests. The gate on a bare scaffold, before any evidence test exists, ends in ERROR (exit code 2): no evidence is not a pass. The quick start tutorial walks the whole path on a small TypeScript project: scaffold, evidence test, PASS, an injected defect, FAIL with checker_failed, and how to read gate.md and gate.json. To write your own specification, see authoring a specification; the scaffold files are described in new project. The witnessbox command needs 0.7.0 or later; upgrade an earlier installation with uv tool upgrade witnessbox. To have a coding agent wire up the project, give it the agent instructions: it drafts the specification and stops until a human approves it. The installation provides one command, witnessbox, with the gate, check, scaffold, sync-pinned and test-checks subcommands; witnessbox --help lists them. The two separate commands from 0.6.0 are removed in 0.7.0. Gate exit codes: 0 PASS, 1 FAIL, 2 ERROR. The machine-readable result for CI is gate.json in the results directory.

witnessbox --installation prints the version and the SHA-256 of the gate and checker code; compare them with the release values in the decision log. Building from source, moving from the previous distribution and migration are covered in usage.

The documentation is written in Russian.

Guarantees and limits

The gate checks what was recorded and re-verifies the rule, but it is not a sandbox: the code under test and the gate run as the same OS user, so the guarantee is detection, not a privilege boundary (D26).

• A decision that was made but not recorded, with consequences not visible outside observation-level (OBS) scenarios, is not caught; mutation audit is a research tool, not part of the gate (D29).
• An installation tampered with before the run is not caught: the caller compares gate_sha256 and checker_sha256 with the release values (D33); in CI, actions/setup checks the wheel and --installation against the release (D45).
• A deliberate bypass from test.sh by the same user — a background process, ~/.gitconfig, PATH — is not caught (D26).
• A coordinated substitution of evidence and gate.json before the verdict is not caught; in CI the trusted stage re-checks the evidence and signs the result, but the evidence is still produced by the code under test (D34, D45).
• In CI, a verdict from the untrusted pull_request stage is not trusted: a workflow_run stage re-verifies the evidence against ebt/ from the base branch, signs gate.json and sets the witnessbox/gate commit status (D45, CI mode).

A human approves the specification (D4). The full "attack → how it is caught → what is not" table is in the threat model.

Documents

• Quick start — a tutorial from installation to the first PASS and a caught defect.
• Authoring a specification — from a requirement to spec.toml, a checker and evidence tests; the observation level (<INV>-OBS) and its contract.
• Instructions for a coding agent — the same path for an AI agent: rules, stop points for human approval, commands and the final report.
• How it works — one example from requirement to verdict, for a first look.
• Threat model — what the gate detects and what it does not promise.
• CI mode — two-stage GitHub Actions gate with a signed verdict and the witnessbox/gate status.
• New project — the ebt/ scaffold: specification, stack-specific test.sh and pinned.txt.
• Architecture — the actual modules and the verification flow.
• Extensions — public types and the contract for project checkers.
• Development — test commands, SDKs and the standalone build.
• Decisions — current status and the append-only decision log.

License

MIT. The SDK files (Ebt.java, ebt.ts, ebt.dart) are copied into application code unchanged and carry the line SPDX-License-Identifier: MIT.