enquiry-bench

An offline benchmark for routing, cited draft replies and abstention on a small fictional service-enquiry task. It scores prediction files you supply. It is not a customer-service agent, makes no model or network calls, and needs no accounts or third-party packages.

Status: 0.1.0 — released 2026-11-05 · github.com/failstocode66/enquiry-bench Exercised on Windows with CPython 3.12.12, 3.13.12 and 3.14.5. Other platforms are unverified.

• Tests: on 2026-11-05 the 36-test suite was run from the release source tree against each of those three interpreters and passed in full each time: 36 passed, with no failures, errors or skips. A run without skips means every link fixture was created, so the native checks ran on real links: a symlinked file, parent directory and output, both aliases of a hard link, and a junction given as input, which was rejected with E_LINK while its target survived. The tests of refusal classification and of product-failure propagation also passed.
• The 2026-11-03 validation run (CPython 3.14.5, 36 passed, no skips) used a non-elevated session (medium integrity, outside an AppContainer, UAC-filtered token). Nothing ran as administrator, and no OS, ACL, security or Developer Mode setting was changed. It shows the link checks working where links can be created. It does not show that the session was unrestricted, and it does not explain the earlier refusal below.
• Earlier revisions: a first 30-test suite passed in full, link tests included. On the 34-test suite that followed, 32 passed, the junction test errored and the hard-link test skipped, both at fixture setup because creating the link was refused (access denied), so that was not a full pass. A standalone check without this package was refused the same way; the cause is not established.
• Packaging: the 0.1.0 archives were built from the release source tree on 2026-11-05 with build 1.6.1 and setuptools 84.2.1. The source distribution carries all 20 project files plus generated metadata; the wheel carries exactly the 10 runtime files plus metadata and license, and declares no third-party dependencies. The wheel's 10 runtime files are byte-identical to those of the wheel validated on 2026-11-03. Installed with --no-index --no-deps into a fresh environment, the exact new wheel ran enquiry-bench demo outside the source tree and produced the figures below, with manifest.txt written. The same two archives are the assets of the v0.1.0 GitHub release, with their SHA-256 sums, and the files published on PyPI.

Install

From PyPI:

pip install enquiry-bench

Or the tagged release straight from the repository:

pip install "git+https://github.com/failstocode66/enquiry-bench@v0.1.0"

Requires Python 3.12 or later; older interpreters are refused with E_RUNTIME rather than run without Windows junction checks.

Quickstart

From a clone of the repository, or after either install above, skip the first line:

python -m pip install --no-deps .
enquiry-bench demo --output demo-out

The demo scores two bundled prediction files, both synthetic: an authored reference that matches the gold, and an always-abstain baseline generated from the case inputs. Neither is output measured from any model. The frozen fixture is designed to give the figures below, so they show the scorer working, not AI-model performance; confirm them in demo-out/summary.md.

Commands

Bundled inputs live in src/enquiry_bench/data/ and install with the package. The source distribution also carries tests/, docs/ and CHANGELOG.md; the wheel does not.

enquiry-bench validate --cases CASES --gold GOLD --knowledge KNOWLEDGE
enquiry-bench run --baseline always-abstain --cases CASES --knowledge KNOWLEDGE --output DIR
enquiry-bench score --cases CASES --gold GOLD --knowledge KNOWLEDGE --predictions PRED --output DIR
    [--repeat-predictions PRED2 ...] [--rates RATES]
enquiry-bench demo --output DIR

run never reads gold; score never executes file content. --output must be a new directory. score writes score.json (byte-reproducible), cases.csv, report.md, confusion.svg and manifest.txt (file hashes, written last). Errors are fixed codes that never echo input text or paths. Python equivalents: validate_files, run_baseline, score_files, demo.

Tests

python -m unittest discover -s tests

Needs the package installed or src on PYTHONPATH. The symlink, hard-link and junction tests create real links. A recognized refusal to create one (link type unsupported, access denied, privilege not held) skips that test with the reason, so its product check did not run and the skip is not a pass. Any other error while creating a link is reported as an error, and once a link exists its product checks run unchanged. No administrator rights, Developer Mode or permission changes are required or made.

What the numbers mean

The headline, joint task pass, needs the right route, the right abstention flag, a consistent route/flag pair, an exact acceptable draft and the exact expected citations. Draft match is exact comparison against a short authored list: a closed-format rubric, not a semantic judge. A reworded correct answer fails, and a citation that exists is not proof the draft is true. Results on this synthetic task say nothing about real-world model performance.

See metrics and schemas, synthetic provenance and the publication post.

Known gaps

• Only Windows has been exercised (CPython 3.12.12, 3.13.12 and 3.14.5); other platforms are unverified.
• Junction and hard-link rejection has been exercised only on Windows, in the sessions above. Wherever link creation is refused those tests skip, and a skip is not a pass. Why creation was refused where it was seen is not established.
• The tests give qualified coverage of path handling, not a proven containment or security guarantee, and path checks have time-of-check races.

License

MIT. Copyright (c) 2026 Tyler Smith. The code and the invented fixtures share this license.