gtrex

Grounded text extraction you can check.

Status: alpha (0.1.0a2). The API and output formats may still change. Known limitations are listed near the end of this file, so that they travel with the package. 0.1.0a2 is a reliability release: see CHANGELOG.md for what changed, and for how to migrate policies and calibrations written by 0.1.0a1.

For fixed extractions, the reported grounding rate depends on the verification policy. "The quotation appears in the source" always involves choices: which normalizations to apply, what to do with an ellipsis, whether stitched fragments count, which tier to call grounded. A rate reported without those choices cannot be compared with anyone else's. gtrex makes the policy explicit, versioned and fingerprinted, so a grounding claim is something another lab can check.

pip install gtrex        # no runtime dependencies, Python >= 3.10

from gtrex import Policy, Verifier

verifier = Verifier(filing_text, Policy())
cert = verifier.verify(model_quote)

cert.grounded   # True
cert.tier       # 'typography'  -- how much normalization it took
cert.start      # 48213         -- offsets into the original text, at every tier
cert.span_text  # the document's own characters

What it does

1. A declarative normalization ladder. Five cumulative tiers, each declaring whether it is sound: a recorded assumption that the tier forgives only presentation, not content, in your domain. It is an assumption, not a proof (case folding merges mW and MW), which is why it travels with the data. Verification reports the tightest tier that contains the whole quotation, not a boolean, and a match must cover whole source characters: 1 taken out of ¼ does not count. On the eight claims of examples/quickstart.py:

tier          at tier   cumulative     share
--------------------------------------------
exact               1            1   12.50%
whitespace          0            1   12.50%
typography          1            2   25.00%
case                1            3   37.50%
skeleton            1            4   50.00%   <- UNSOUND, not counted
stitched            1            -   12.50%
unlocated           3            -   37.50%

The skeleton tier is marked unsound because the collision is real, not hypothetical:

"Operating loss was ($10.0 million)."  ->  operatinglosswas100million
"Operating loss was $100 million."     ->  operatinglosswas100million

Accepting it requires passing allow_unsound=True explicitly.

2. A repair-and-reject cascade. When a quotation is a near-miss, the cascade (in its default substitute mode) does not accept the model's wording: it locates the source span and substitutes the document's own characters, keeping the model's original wording and the certificates before and after. Anything that cannot be located above the threshold you set is rejected. annotate mode keeps the model's wording and attaches the located span as evidence; off only verifies.

verify -> [fragment salvage, opt-in] -> locate-and-substitute -> reject

Verification always checks the whole claim; nothing is clipped before it is compared, so a verbatim opening with an invented tail is not grounded. A substitution may then store a source span shorter than the claim: the stored text is certified by its own after certificate, and the claim as given keeps its before certificate, which says it did not verify.

3. Two guards for edits no threshold can catch. A changed figure and a dropped negation both score high, because they share almost every token with the source, and both would yield a record that is verbatim source text and still does not say what the model extracted. From the quickstart:

claim: "Operating loss was $100 million for the year ended ..."    score 0.87
span:  "Operating loss was ($10.0 million) for the year ended ..." -> numeric_guard

claim: "We believe that we will be able to compete effectively ..."     score 0.97
span:  "We believe that we will not be able to compete effectively ..." -> polarity_guard

These are checks, not weights, and they are partial (see the limitations below): they block two common catastrophic edits; they do not prove that a substitution preserves meaning.

4. A threshold you calibrate, not guess. The substitution threshold has no default, and repairing without one is an error. gtrex calibrate estimates one with Learn-then-Test from labelled repairs, states the assumption the guarantee rests on (see "Guarantees, precisely" below), and binds the result to the scorer and pipeline it was made for, so that gtrex repair --calibration refuses to use it anywhere else.

5. A manifest. Policy, fingerprints, package version, Unicode database version, every tier's soundness rationale, calibration, and the ladder profile, written next to the data so the assumptions travel with it. Every part must come from the manifest's own policy; mixing is refused.

Why a ladder instead of a number

Three failure modes a single grounding rate cannot distinguish:

A verifier that reports one number will call the first two "grounded" and the third "hallucinated". All three verdicts would be wrong.

CLI

gtrex explain                       # print the policy and every assumption
gtrex verify   claims.jsonl -o certified.jsonl --manifest run.json
gtrex calibrate labelled.jsonl -o calibration.json --alpha 0.05 --compare 0.65
gtrex repair   claims.jsonl -o repaired.jsonl --calibration calibration.json --drop-rejected
gtrex sensitivity claims.jsonl      # the same claims under three policies

repair needs a threshold: --calibration takes the one calibrate wrote, after checking it was calibrated for this policy's scorer and pipeline, and --threshold sets one by hand. --repair-mode annotate keeps the model's wording; --salvage longest_grounded opts into fragment salvage.

Input is JSON Lines. Each record needs a string claim and either an inline source or a source_path (read as UTF-8, byte for byte). Every other field is passed through untouched and in input order; the result is added under one key, gtrex. A record that already has a gtrex field is refused unless you pass --replace-existing.

{"id": "rf-04", "claim": "We depend on a limited number of suppliers", "source_path": "filings/acme-2023/item1a.txt"}

gtrex repair stores the record's text in claim (in substitute mode, the source's own text), and keeps the model's wording in gtrex.claim_original, with gtrex.before (the certificate of the claim as given) and gtrex.after (the certificate of the stored text, when it differs). gtrex.accepted says the record was retained; gtrex.grounded says its text verifies. An annotation is accepted and not grounded.

Exit status is 0 on success and 1 on invalid input (the message names the file and line) or a failed calibration; JSON goes to stdout only when stdout is the requested output, and is always UTF-8, whatever the console encoding.

sensitivity is the experiment worth running first on your own data. On the quickstart's eight claims:

policy       fingerprint      accept_through   grounded  contiguous
--------------------------------------------------------------------
default      826164faf91a51c5 case              37.50%     37.50%
strict       154596cf2cd758f8 exact             12.50%     12.50%
permissive   6e9bd2488a1bd1c1 skeleton          50.00%     50.00%

Same 8 claims, same sources. The reported grounding rate varies by 37.5
percentage points across these three verification policies.

grounded is the share of claims the policy accepts, the mean of the individual verdicts; contiguous is the share contained as one run through the accepted tier. They differ only when a policy accepts stitched quotations. The three policies differ in more than normalization (ellipsis handling and contiguity too); gtrex explain --policy strict prints each one in full.

Guarantees, precisely

A repair is accepted when score >= threshold, and calibration uses the same rule. calibrate_risk implements Learn-then-Test with fixed-sequence testing over the observed scores (plus 1.0, or a grid you supply), using the exact binomial tail as the p-value, and returns the least conservative certifiable threshold among those candidates. The controlled quantity is the joint risk

P(score >= threshold AND the repair is unfaithful)

for claims like the calibration claims that reach the alignment-score filter. It is not P(unfaithful | accepted), which is reported only as a point estimate, and it does not cover claims accepted at verification or by fragment salvage. The guarantee, P(risk <= alpha) >= 1 - delta over the calibration draw (delta is the failure probability; the confidence is 1 - delta), holds only if the calibration examples are independent draws from the claims you will repair. Several examples from one document, or several perturbations of one quotation, are not independent: give each example a group (document id) and calibrate_risk refuses repeated groups. Overriding that is recorded in the calibration as uncertified.

calibrate_coverage is split conformal on 1 - similarity, giving P(a genuine repair is accepted) >= 1 - alpha for exchangeable scores at the same stage; the guards may still block some of those repairs afterwards.

A calibration records its target, stage, assumptions, a digest of its data, and the identity of the scorer and pipeline its scores came from; a manifest or repair --calibration refuses one made for another configuration. A threshold calibrated on synthetic perturbations is calibrated for those perturbations. Use the certified value at full precision (scores and thresholds are exported unrounded); rounding it down admits scores the certificate excluded. Both functions raise CalibrationError rather than return an uncertifiable threshold.

What it does not check

• Entailment. A verbatim quotation can be attached to the wrong entity, relation, or direction. Grounding is necessary, not sufficient.
• Recall. Nothing here measures what the model failed to extract.
• Attribute fields. Verify the quotation, then validate your schema's other fields against it separately.

A rejected claim is not a detected hallucination. It is a claim the policy will not vouch for, which also covers splicing, reordering, paraphrase, parser drift, and a source that was not the document the model saw.

Known limitations

This is an alpha. Before relying on it, know that:

• Containment is literal substring matching. Revenue was $10 is a substring of Revenue was $100 million, and is grounded. Matching does not respect word or number boundaries.
• "Sound" is an assumption. A sound tier is one the policy declares to forgive only presentation. Case folding merges mW and MW; typography folding merges a minus sign with a hyphen. Choose the ladder for your domain.
• The guards are partial. The numeric guard compares numbers read by the English convention (, groups thousands, . marks decimals; a malformed number, or one written with a fraction character such as 1½, must match verbatim; a superscript is a separate figure), so it ignores units and scale words ($1.5 million vs $1.5 thousand), sign conventions, figure order and spelled-out numbers. The polarity guard counts watched words, so it misses contractions (don't), a negation moved within a sentence, and direction words (increased vs decreased); it also counts the month "May" as a modal.
• The fingerprint covers the configuration, including every normalization step's id, version and parameters, plus a verifier revision that is bumped when verdicts change. It is not a hash of the code: report the package version with it.
• Unicode. Normalization is per character, so a decomposed accent does not match its precomposed form. Verdicts depend on the Python's Unicode database, which the manifest records; the same fingerprint under another database is not a claim of identical verdicts. Tokenization assumes space-separated scripts.
• Calibration is synthetic until you label your own repairs. The guarantees hold for the population the calibration examples were drawn from, at the alignment stage only.
• Performance. Each source is normalized once per tier; very large corpora are slower than they need to be.
• Repair. Substitution trusts the aligner's best span above your threshold, and may store a span shorter than the claim. Keep gtrex.claim_original and review substituted records where it matters. Opted-in salvage keeps one verified fragment and records what it dropped; what was dropped is not certified.

Documentation

docs/DECISIONS.md (in the source distribution and repository) documents every parameter: what it assumes, what it costs, and when to change it. Read it before reporting a number. examples/quickstart.py is a self-contained tour.

Development

From a source checkout:

pip install -e ".[dev]"
pytest

Citation

If you use this in published work, report your Policy.fingerprint, the package version and the Unicode database version (all in the manifest) alongside any grounding rate.

License

Apache-2.0.