gtrex

Grounded text extraction you can check.

gtrex checks whether a quotation extracted by a language model appears in its source text, under a verification policy that is explicit, versioned and fingerprinted. It certifies literal quotation provenance. It does not certify that an extracted fact is true.

pip install gtrex        # Python >= 3.10, no runtime dependencies

Verify

from gtrex import Policy, Verifier

verifier = Verifier(source_text, Policy())
cert = verifier.verify(model_quote)

cert.grounded   # whether the policy accepts the quotation
cert.tier       # how much normalization it took, e.g. 'typography'
cert.start      # character offsets into the original source text
cert.span_text  # the source's own characters

Verification always checks the whole quotation, so a verbatim opening with an invented tail is not grounded. If your extraction stores a source location, verifier.verify_at(model_quote, start, end) checks that location only: a match elsewhere in the document cannot validate it. The certificate then reports scope="span", the requested_span, and, after a failed check, whether a contiguous match exists elsewhere.

Policies

A grounding rate depends on choices: which normalizations to apply, what to do with an ellipsis, whether stitched fragments count, and which tier counts as grounded. A Policy records all of them. Policy.fingerprint digests that configuration together with a verifier revision, which changes whenever the code changes a verdict. Report the fingerprint, the package version and the Unicode database version with any grounding rate; the run manifest records all three.

The default ladder has five cumulative tiers: exact, whitespace, typography, case and skeleton. Each tier declares whether it is sound, an assumption that it forgives presentation, not content. Verification reports the tightest tier that contains the whole quotation, and a match must cover whole source characters. skeleton is unsound, because removing punctuation makes different statements collide:

"Operating loss was ($10.0 million)."  ->  operatinglosswas100million
"Operating loss was $100 million."     ->  operatinglosswas100million

Counting an unsound tier as grounded requires allow_unsound=True.

Three policies are built in:

• STRICT accepts only character-identical quotations.
• DEFAULT accepts the sound tiers. A quotation joined from separate fragments with an ellipsis is reported as stitched, not grounded.
• PERMISSIVE accepts the skeleton tier, ignores ellipses and admits stitched fragments. It exists to measure how much such a policy inflates a rate, not as a recommendation.

gtrex explain --policy strict prints a policy with every assumption and its fingerprint. If an upstream parser inserted text the model never saw, such as layout markers, genuine quotations fail; declare it with marker_pattern.

Repair

A Repairer handles near-misses:

verify -> [fragment salvage, opt-in] -> locate-and-substitute -> reject

In the default substitute mode it locates the best-matching source span and stores the source's own characters, keeping the model's wording and the certificates before and after. annotate keeps the model's wording and attaches the located span; off only verifies. The substitution threshold has no default: calibrate one or set one, because repairing without one is an error. A substitution may store a source span shorter than the claim; the stored text is certified by its own after certificate.

Two guards refuse substitutions that no similarity threshold can catch, because the edited text shares almost every token with the source:

claim: "Operating loss was $100 million for the year ended ..."
span:  "Operating loss was ($10.0 million) for the year ended ..."       -> numeric_guard

claim: "We believe that we will be able to compete effectively ..."
span:  "We believe that we will not be able to compete effectively ..."  -> polarity_guard

The numeric guard compares figures, their order and adjacent signs. The polarity guard compares negation, modal and direction words (not, may, will, increase, decrease, more, less, higher, lower and others) in the claim and the located span. Both are partial checks: they block common meaning-changing edits; they do not prove that a substitution preserves meaning.

Command line

gtrex explain                        # the policy and every assumption
gtrex verify    claims.jsonl -o certified.jsonl --manifest run.json
gtrex repair    claims.jsonl -o repaired.jsonl --calibration calibration.json
gtrex calibrate labelled.jsonl -o calibration.json --alpha 0.05
gtrex sensitivity claims.jsonl       # the same claims under the three built-in policies
gtrex audit-export answers.csv -o audit.json

Input is JSON Lines. Each record needs a string claim and exactly one of an inline source or a source_path (read as UTF-8, byte for byte):

{"id": "q1", "claim": "We depend on a limited number of suppliers", "source_path": "docs/report.txt"}

Other fields pass through unchanged and in input order, and results go under one key, gtrex. A record that already has a gtrex field is refused unless you pass --replace-existing. verify --at-offsets requires start and end in every record and checks only that span.

repair needs a threshold: --calibration uses the one calibrate wrote, after checking that it was made for this policy's scorer and pipeline, and --threshold sets one by hand. It stores the record's text in claim (in substitute mode, the source's own text) and keeps the model's wording in gtrex.claim_original, with the certificates gtrex.before and gtrex.after. gtrex.accepted says the record was kept; gtrex.grounded says its text verifies. An annotation is accepted and not grounded.

calibrate reads JSON Lines of {"score": ..., "faithful": true, "group": ...}, where score is a repair's gtrex.alignment.score and faithful is your judgement of that repair. sensitivity reports, for each built-in policy, the share of claims it accepts and the share contained as one contiguous run.

audit-export checks that a CSV export kept an evidence value for every populated answer column (by default Q... columns paired with Q..._evidence) and exits nonzero if one is missing. It checks completeness only, not the evidence itself.

Exit status is 0 on success and 1 on invalid input (the message names the file and line), a failed calibration or an incomplete export. Output is always UTF-8.

Calibration, precisely

A repair is accepted when score >= threshold, and calibration uses the same rule. calibrate_risk implements Learn-then-Test with fixed-sequence testing over the observed scores (plus 1.0, or a grid you supply), using the exact binomial tail as the p-value, and returns the least conservative certifiable threshold among those candidates. The controlled quantity is the joint risk

P(score >= threshold AND the repair is unfaithful)

for claims like the calibration claims that reach the alignment-score filter. It is not P(unfaithful | accepted), which is reported only as a point estimate, and it does not cover claims accepted at verification or by fragment salvage. The guarantee, P(risk <= alpha) >= 1 - delta over the calibration draw, holds only if the calibration examples are independent draws from the claims you will repair. Several examples from one document, or several perturbations of one quotation, are not independent: give each example a group (such as a document id), and calibrate_risk refuses repeated groups. Overriding that is recorded in the calibration as uncertified.

calibrate_coverage is split conformal on 1 - similarity, giving P(a genuine repair is accepted) >= 1 - alpha for exchangeable scores at the same stage; the guards may still block some of those repairs afterwards.

A calibration records its target, stage, assumptions, a digest of its data, and the identity of the scorer and pipeline its scores came from. A manifest or repair --calibration refuses a calibration made for another configuration, and the loader rejects incompatible method revisions, acceptance rules, targets, stages and unknown fields. This validates the file's declared format; it does not protect it against deliberate editing. A threshold calibrated on synthetic perturbations is calibrated for those perturbations. Use the certified value at full precision (scores and thresholds are exported unrounded), because rounding it down admits scores the certificate excluded. Both functions raise CalibrationError rather than return an uncertifiable threshold.

What it does not check

• Entailment. A verbatim quotation can be attached to the wrong entity, relation or direction. Grounding is necessary, not sufficient.
• Recall. Nothing here measures what the model failed to extract.
• Other fields. Verify the quotation, then validate your schema's other fields against it separately.

A rejected claim is not a detected hallucination. It is a claim the policy will not vouch for, which also covers splicing, reordering, paraphrase, parser artifacts, and a source that was not the document the model saw.

Known limitations

• Lexical boundaries are heuristic. By default a match may not start or end inside a word, hyphenated term, signed figure, decimal, grouped number or time. HTML entities are decoded before the edges are checked, and apostrophe-suffixed names and sentence punctuation are allowed. Scripts that do not separate words with spaces need a language-specific tokenizer. match_boundaries=False turns the check off.
• "Sound" is an assumption. Case folding merges mW and MW, and typography folding merges a minus sign with a hyphen. Choose the ladder for your domain.
• The guards are partial. The numeric guard reads numbers by the English convention (, groups thousands, . marks decimals). A malformed number, or one written with a fraction character such as 1½, must match verbatim, and a superscript is a separate figure. It ignores physical units, uncommon scale words, many sign conventions and spelled-out numbers. The polarity guard also reads negative words just before the located span. It misses some shifted negation scopes, it counts the month "May" as a modal, and it can refuse a valid repair because of a negation earlier in the sentence, such as in a preceding clause or table row. The guards run on alignment repair, not on claims that already verify as literal source text.
• The fingerprint covers the configuration, including every normalization step's id, version and parameters, plus the verifier revision. It is not a hash of the code, so report the package version with it.
• Unicode. Normalization is per character, so a decomposed accent does not match its precomposed form. Verdicts depend on Python's Unicode database, which the manifest records; the same fingerprint under another database is not a claim of identical verdicts.
• Calibration is synthetic until you label your own repairs. The guarantees hold for the population the calibration examples were drawn from, at the alignment stage only.
• Repair. Substitution trusts the aligner's best span above your threshold. Keep gtrex.claim_original, and review substituted records where it matters. Opted-in salvage keeps one verified fragment and records what it dropped; what was dropped is not certified.
• Performance. Each source is normalized once per tier, so very large corpora are slower than they need to be.

Upgrading from a pre-release

Policy files written by 0.1.0b1 to 0.1.0b4 load unchanged, with their own fingerprints. Files from earlier pre-releases load through an explicit migration that warns and assigns a new fingerprint. In 1.0.0 the default policies also watch "higher" and "lower", so the built-in fingerprints changed. A calibration stays bound to the policy it was made for: use it with that policy's file, or recalibrate. The repository includes the 0.1.0b4 default policy as tests/fixtures/gtrex-0.1.0b4-policy.json for reproducing that beta's calibrated decisions. Its stored fingerprint is 63c0146e38cdb3cd; the 1.0.0 built-in DEFAULT has a different fingerprint.

License

Apache-2.0.