evidentia-eval

Dev-time AI-output quality eval harness for Evidentia.

Hosts the DFAH (Decision-Faithfulness Assessment Harness) — the auditor-defensible numerical proof layer that validates LLM-driven artifact production is deterministic, replay- equivalent, and faithful to its source policy clauses.

Why this package exists (v0.10.5 P9 extraction)

The DFAH harness was originally bundled into evidentia-ai (the risk-statement generator + control explainer package). That conflated two very different deployment surfaces:

evidentia-ai — PRODUCTION runtime. Needed in air-gap installs to actually generate risk statements.
evidentia-eval — DEVELOPMENT-time evaluation. NOT needed in air-gap installs; only fires when a CI pipeline runs a determinism / faithfulness gate before tagging a release.

Extracting the eval harness lets air-gap installs of evidentia-ai skip the optional sentence-transformers stack entirely (it now lives behind evidentia-eval[faithfulness-semantic] instead of evidentia-ai[eval-faithfulness]).

Quick start

# Stdlib Jaccard baseline (no extra needed; <10 MB install)
pip install evidentia-eval

# Optional semantic-similarity faithfulness (~250 MB extra
# for sentence-transformers + numpy + model cache on first use)
pip install 'evidentia-eval[faithfulness-semantic]'

CLI verbs:

# Smoke test against a deterministic stub generator (no LLM
# tokens burned)
evidentia eval stub-smoke

# Real-LLM determinism gate against the risk-statement generator
evidentia eval risk-determinism --gap-report gaps.json \
    --system-context ctx.yaml \
    --fail-on-determinism-rate-below 0.95

# Verify a previously-signed eval bundle
evidentia eval verify path/to/eval-output.json

The CLI verbs live in evidentia.cli.eval (the meta-package); this package contributes the underlying library.

Public API

Symbol	Purpose
`DFAHarness`	Owns the run loop + audit emit
`EvalResult`	Top-level harness output (JSON-serializable, Sigstore-signable)
`EvalSample`	One prompt's inputs (immutable; audit-trail-stable)
`DeterminismResult`	Per-prompt determinism outcome
`ReplayResult`	Per-prompt replay-equivalence outcome
`FaithfulnessResult`	Per-claim faithfulness outcome
`PromptFaithfulnessResult`	Aggregated per-prompt faithfulness
`faithfulness_score`	Stdlib Jaccard token-overlap baseline
`faithfulness_score_semantic`	Sentence-transformers path (optional extra)
`determinism_score`	Computes the modal-output pass rate
`replay_equivalent`	Binary replay-equivalence check
`extract_claims`	Atomic-claim extraction from generated artifacts
`normalize_for_determinism`	Canonical whitespace + punctuation normalization
`hash_output`	SHA-256 hex of normalized output
`sign_eval_result`	Sigstore-sign an `EvalResult` JSON
`verify_eval_result`	Verify a previously-signed eval bundle

Migrating from `evidentia_ai.eval`

The v0.10.5 P9 extraction left evidentia_ai.eval.* in place as re-export shims that warned once at import time. They were removed in v0.12.0 as announced, together with the unconditional evidentia-eval dependency on evidentia-ai. Replace every from evidentia_ai.eval import X with from evidentia_eval import X — same symbols, same signatures.

The evidentia-ai[eval-faithfulness] install extra still works; only the import path is gone.

License

Apache-2.0. See the workspace root LICENSE file.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

evidentia_eval-0.12.0.tar.gz (27.1 kB view details)

Uploaded Aug 23, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

evidentia_eval-0.12.0-py3-none-any.whl (30.1 kB view details)

Uploaded Aug 23, 2026 Python 3

File details

Details for the file evidentia_eval-0.12.0.tar.gz.

File metadata

Download URL: evidentia_eval-0.12.0.tar.gz
Upload date: Aug 23, 2026
Size: 27.1 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for evidentia_eval-0.12.0.tar.gz
Algorithm	Hash digest
SHA256	`e821247a86b287cbbd569d19714f24518b1f94c2979065179b1b7bff36f55f18`
MD5	`aad5dc9b1a1c6a5a940a6fab342b69d7`
BLAKE2b-256	`f5d457e308d8b667dc2b061a208fa789c19594fe4b7d614eccc030dcf605560b`

See more details on using hashes here.

Provenance

The following attestation bundles were made for evidentia_eval-0.12.0.tar.gz:

Publisher: release.yml on Polycentric-Labs/evidentia

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: evidentia_eval-0.12.0.tar.gz
- Subject digest: e821247a86b287cbbd569d19714f24518b1f94c2979065179b1b7bff36f55f18
- Sigstore transparency entry: 2578527214
- Sigstore integration time: Aug 23, 2026
Source repository:
- Permalink: Polycentric-Labs/evidentia@b7ac58432e65d3727b01a797bf01290b9799356d
- Branch / Tag: refs/tags/v0.12.0
- Owner: https://github.com/Polycentric-Labs
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@b7ac58432e65d3727b01a797bf01290b9799356d
- Trigger Event: push