deepeval-openttt

TTTPS Proof-of-Time provenance metric for DeepEval. It answers one question about a test case: does this output carry a genuine Proof-of-Time receipt that actually covers it, according to the public self-serve KPP Provenance API?

Deterministic and rule based. There is no LLM judge, no model call, and no API key needed for scoring existing receipts.

This attaches and checks a cryptographic audit-trail timestamp and integrity hash. It does not certify legal or regulatory compliance (EU AI Act, FDA, and so on). Treat it as an audit-trail timestamp, not a compliance claim.

Install

pip install deepeval-openttt

Usage

Score a receipt your pipeline already produced, for example one attached by litellm-openttt, llamaindex-openttt or smolagents-openttt:

from deepeval.test_case import LLMTestCase
from deepeval_openttt import TTTPSProvenanceMetric

test_case = LLMTestCase(
    input="how tall is the Eiffel Tower?",
    actual_output="The Eiffel Tower is 330 metres tall.",
    metadata={"tttps_receipt": receipt},   # raw KPP receipt dict
)

metric = TTTPSProvenanceMetric()
metric.measure(test_case)
print(metric.score, metric.reason)
# 1.0  Structurally valid receipt (id=...). cross-checked against the KPP Provenance API (...)

Or let the metric seal the output itself (needs a key, mint one for free with POST https://kpp.kenosian.com/v1/keys):

metric = TTTPSProvenanceMetric(anchor_if_missing=True)   # reads KPP_API_KEY

Both evaluate([...], [metric]) and assert_test work, since this is a plain BaseMetric with measure() and a_measure().

What it checks, and what it cannot

Two layers. The structural layer asks "is a well formed receipt attached". The cross-check layer, on by default, asks "does the API still have this record, and does it cover this exact output":

1. content binding: sha256(actual_output) equals the anchored digest
2. existence: the API returns a verified record for this receipt id
3. field agreement: the receipt's digest and timestamp equal the API's

Measured 2026-08-05 over 25 rows of the published corpus Pittro/verifiable-ai-provenance-bench plus 5 negatives built by mutating real receipts:

Negatives caught: 1 of 5 structure only, 5 of 5 with the default. Separation between the genuine mean and the negative mean: 0.20 structure only, 1.00 with the default.

Reproducing those rows needs only your own key: anchor a text, then (a) attach the receipt to a different text, (b) flip one hex character of content_hash, (c) rewrite time by a year, (d) invent a 24 hex character receipt id, (e) attach nothing.

What it does not tell you:

• nothing about whether the output is correct, safe or useful. It is a provenance check, not a quality check
• a receipt proves the content existed no later than the anchored time. It cannot prove the content did not exist earlier
• if your pipeline anchors a different byte string than actual_output (a canonical JSON, a trimmed answer), content binding fails. Anchor the exact string you evaluate, or set bind_content=False and accept that a lifted receipt then scores 1.0
• with cross_check=False it detects nothing beyond a missing or malformed receipt
• it does not require any particular time source. The API reports time_source (roughtime_chain or local) on the record kept in metric.tttps_server_record, so gate on that yourself if chain backed time matters to you

Score semantics

Per case the score is binary, 1.0 or 0.0: a receipt either covers this output or it does not. The continuous number a DeepEval report shows for a suite is the mean over N cases, in other words a pass rate, not a per case confidence.

Options

Environment: KPP_BASE (default https://kpp.kenosian.com), KPP_API_KEY, KPP_TIMEOUT_S (seconds, default 1.0).

Fail-open: a slow or unreachable API degrades the score and records why in metric.reason. It never raises out of the eval run. With cross_check=True the metric makes one POST /v1/verify round trip per test case.

License

MIT