ragas-openttt

TTTPS Proof-of-Time provenance metric for Ragas. It answers one question about a sample: does this response have a genuine Proof-of-Time anchor on the public self-serve KPP Provenance API, and does that anchor cover this exact response?

Deterministic and rule based, in the same family as ragas' own ExactMatch and StringPresence. There is no LLM judge and no model call.

This checks a cryptographic audit-trail timestamp and integrity hash. It does not certify legal or regulatory compliance (EU AI Act, FDA, and so on). Treat it as an audit-trail timestamp, not a compliance claim.

Install

pip install ragas-openttt

The dependency list pins langchain-community<0.4.2 on purpose. ragas 0.4.3 imports langchain_community.chat_models.vertexai unconditionally at import time and langchain-community removed that module in 0.4.2, so without the bound import ragas raises ModuleNotFoundError on a fresh install. Measured 2026-08-05: the module is present in 0.4 and 0.4.1, absent from 0.4.2. Drop the bound once ragas no longer needs it.

Usage

The receipt id travels in retrieved_context_ids, as a one element list. SingleTurnSample is a closed pydantic model with no free metadata field, and a subclass that adds one is rejected by ragas' strict sample type check, so this is the one slot that keeps a sample usable by real ragas.evaluate().

from ragas import evaluate
from ragas.dataset_schema import EvaluationDataset, SingleTurnSample
from ragas_openttt import TTTPSProvenanceMetric

sample = SingleTurnSample(
    user_input="how tall is the Eiffel Tower?",
    response="The Eiffel Tower is 330 metres tall.",
    retrieved_context_ids=[receipt_id],       # from POST /v1/anchor
)

result = evaluate(EvaluationDataset(samples=[sample]), metrics=[TTTPSProvenanceMetric()])
print(result)

To seal responses that have no receipt yet, pass a key (mint one for free with POST https://kpp.kenosian.com/v1/keys):

TTTPSProvenanceMetric(anchor_if_missing=True, kpp_key="...")   # or KPP_API_KEY

What it checks, and what it cannot

The API is the source of truth. Given a receipt id the metric calls POST /v1/verify, requires verified: true, and with bind_content=True (the default) requires the anchored digest to equal sha256(sample.response).

That design has a consequence worth stating plainly: the metric never reads a receipt blob held by your pipeline, so edits to that blob are invisible to it. If the anchor on the server is intact, the metric says so, even when the receipt JSON you display to a user has been rewritten.

Measured 2026-08-05 over 25 rows of the published corpus Pittro/verifiable-ai-provenance-bench plus 5 negatives built by mutating real receipts:

Negatives caught by the metric: 2 of 5 without content binding, 3 of 5 with the default. Separation between the genuine mean and the negative mean with the default: 0.60.

The two misses are the two blob edits, for the reason above: the server copy of both receipts is genuine, so the server says verified. If your pipeline keeps or displays the receipt JSON, close that gap with the helper this package ships, which reads the blob and caught 5 of 5 in the same run:

from ragas_openttt import check_receipt_blob

ok, reason = check_receipt_blob(receipt_dict, response_text)

It checks content binding, existence, digest agreement and timestamp agreement, and returns (ok, reason).

Reproducing those rows needs only your own key: anchor a text, then (a) attach the receipt to a different text, (b) flip one hex character of content_hash, (c) rewrite time by a year, (d) invent a 24 hex character receipt id, (e) attach nothing.

What none of it tells you: whether the response is correct, safe or useful. A receipt proves the content existed no later than the anchored time, and never that it did not exist earlier.

Score semantics

Per sample the score is binary, 1.0 or 0.0. The continuous number a Ragas report shows for a dataset is the mean over N samples, in other words a pass rate, not a per sample confidence.

Options

Set bind_content=False only for corpora that publish digests without the plaintext they cover. With bind_content=True and no response text the metric scores 0.0 rather than silently skipping the check.

Fail-open: a slow or unreachable API degrades the score and records why in metric._last_reason. Scoring never raises out of the eval run. The metric makes one POST /v1/verify round trip per sample.

License

MIT