evidentia-eval

Dev-time AI-output quality eval harness for Evidentia.

Hosts the DFAH (Decision-Faithfulness Assessment Harness) — the auditor-defensible numerical proof layer that validates LLM-driven artifact production is deterministic, replay- equivalent, and faithful to its source policy clauses.

Why this package exists (v0.10.5 P9 extraction)

The DFAH harness was originally bundled into evidentia-ai (the risk-statement generator + control explainer package). That conflated two very different deployment surfaces:

  • evidentia-ai — PRODUCTION runtime. Needed in air-gap installs to actually generate risk statements.
  • evidentia-eval — DEVELOPMENT-time evaluation. NOT needed in air-gap installs; only fires when a CI pipeline runs a determinism / faithfulness gate before tagging a release.

Extracting the eval harness lets air-gap installs of evidentia-ai skip the optional sentence-transformers stack entirely (it now lives behind evidentia-eval[faithfulness-semantic] instead of evidentia-ai[eval-faithfulness]).

Quick start

# Stdlib Jaccard baseline (no extra needed; <10 MB install)
pip install evidentia-eval

# Optional semantic-similarity faithfulness (~250 MB extra
# for sentence-transformers + numpy + model cache on first use)
pip install 'evidentia-eval[faithfulness-semantic]'

CLI verbs:

# Smoke test against a deterministic stub generator (no LLM
# tokens burned)
evidentia eval stub-smoke

# Real-LLM determinism gate against the risk-statement generator
evidentia eval risk-determinism --gap-report gaps.json \
    --system-context ctx.yaml \
    --fail-on-determinism-rate-below 0.95

# Verify a previously-signed eval bundle
evidentia eval verify path/to/eval-output.json

The CLI verbs live in evidentia.cli.eval (the meta-package); this package contributes the underlying library.

Public API

Symbol Purpose
DFAHarness Owns the run loop + audit emit
EvalResult Top-level harness output (JSON-serializable, Sigstore-signable)
EvalSample One prompt's inputs (immutable; audit-trail-stable)
DeterminismResult Per-prompt determinism outcome
ReplayResult Per-prompt replay-equivalence outcome
FaithfulnessResult Per-claim faithfulness outcome
PromptFaithfulnessResult Aggregated per-prompt faithfulness
faithfulness_score Stdlib Jaccard token-overlap baseline
faithfulness_score_semantic Sentence-transformers path (optional extra)
determinism_score Computes the modal-output pass rate
replay_equivalent Binary replay-equivalence check
extract_claims Atomic-claim extraction from generated artifacts
normalize_for_determinism Canonical whitespace + punctuation normalization
hash_output SHA-256 hex of normalized output
sign_eval_result Sigstore-sign an EvalResult JSON
verify_eval_result Verify a previously-signed eval bundle

Backward-compat shim

For external scripts that still import from evidentia_ai.eval import ..., evidentia-ai ships a deprecation shim that re-exports from evidentia_eval. The shim warns once at import time and is scheduled for removal in v0.12.0.

License

Apache-2.0. See the workspace root LICENSE file.

Project links

Data verified by PyPI on Aug 17, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for Qd9drCBe from gravatar.com Qd9drCBe

Credits

PyPI data
Data sourced directly from PyPI's database.

Author: Allen Byrd

GitHub Statistics

Data verified by PyPI on Aug 17, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

Apache-2.0
View SPDX License List

Requires

Python <3.14, >=3.12

Provides Extra

faithfulness-semantic

Tags

ai-quality compliance determinism dfah faithfulness grc llm-eval

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

evidentia_eval-0.11.2.tar.gz (26.9 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

evidentia_eval-0.11.2-py3-none-any.whl (30.0 kB view details)

Uploaded Python 3

File details

Details for the file evidentia_eval-0.11.2.tar.gz.

File metadata

  • Download URL: evidentia_eval-0.11.2.tar.gz
  • Upload date:
  • Size: 26.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for evidentia_eval-0.11.2.tar.gz
Algorithm Hash digest
SHA256 b00a763ad627cb3d03c6cebd0eb2943caf13c8b5dfc74eb358d2ffd771ee30a8
MD5 d302c0141f6e8d600286817267221154
BLAKE2b-256 6076afc066924a0a064a7061ef66296c175c252b187873b5a64130296223b937

See more details on using hashes here.

Provenance

The following attestation bundles were made for evidentia_eval-0.11.2.tar.gz:

Publisher: release.yml on Polycentric-Labs/evidentia

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file evidentia_eval-0.11.2-py3-none-any.whl.

File metadata

  • Download URL: evidentia_eval-0.11.2-py3-none-any.whl
  • Upload date:
  • Size: 30.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for evidentia_eval-0.11.2-py3-none-any.whl
Algorithm Hash digest
SHA256 6393e7a9d72cf3a47dd92c210e48508e819e4868eabc54f6d56827e69b1f2f6c
MD5 4b768822ff178184df66033e3cb1e048
BLAKE2b-256 f3f63de46fef9df9aa2795ca29632624c6e9bbbee969bbdcd7e54498a894d279

See more details on using hashes here.

Provenance

The following attestation bundles were made for evidentia_eval-0.11.2-py3-none-any.whl:

Publisher: release.yml on Polycentric-Labs/evidentia

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page