VerifyDoc

The trust layer for document → structured-JSON extraction. Wrap any extractor — get back JSON where every field carries a calibrated confidence, a source grounding (page + bbox / char span), and an accept/review decision tuned to your error budget.

Above: a real pipeline run (scripts/make_demo_gif.py). The extractor returned $1,432.50; the page says $1,234.50. Grounding support drops to 0.78, the field misses the accept threshold, and the reviewer is pointed at the exact source region. The other three fields are auto-accepted.

The problem

Modern document parsers read pages at 96%+ benchmark accuracy — and still emit fluent, plausible, silently-wrong values ($42.50 → $45.20) with no reliable per-field signal telling you which values to trust. Commercial APIs (Box, Azure, Textract) sell field-level confidence as a closed feature. No popular open-source parser leads with it. (full USP audit)

VerifyDoc doesn't compete with the parsers — it layers on top of any of them:

document + schema ─► ingest ─► extractor adapter ─► confidence ─► calibration
                                (any model)          signals       (fit on cal split)
                     ─► grounding ─► abstention policy ─► verified JSON + review UI
                        (bbox/span)   (target risk α)

At a chosen operating point, VerifyDoc auto-accepts as many fields as possible while holding the error rate among accepted fields below your target (e.g. ≤ 2%) — everything else is routed to review with its source location attached, so a human verifies in seconds instead of eyeballing every field.

Quickstart

pip install verifydoc          # core (text pipelines + eval harness)
pip install 'verifydoc[pdf]'   # + PDF/image ingestion

from verifydoc import verify

# ready-to-run sample lives in examples/
result = verify("examples/invoice.txt", schema="examples/invoice_schema.json")
for f in result.fields:
    print(f"{f.path:12} = {f.value!r:24} conf={f.confidence:.2f} {f.decision}")
    if f.grounding:
        print(f"             └─ page {f.grounding.page}, bbox {f.grounding.bbox}")

verifydoc extract examples/invoice.txt --schema examples/invoice_schema.json --threshold 0.8
streamlit run ui/streamlit_app.py     # review UI: green/red fields + click-through to source

See examples/ for the full runnable walk-through.

For AI agents (MCP)

Give any MCP-capable agent (Claude Desktop, IDEs, custom agents) a trust layer for reading documents — so it acts on confident, grounded fields and escalates the rest instead of hallucinating forward:

pip install 'verifydoc[mcp]'
verifydoc-mcp     # stdio MCP server exposing verify_extraction()

Every field the agent extracts comes back with confidence + grounding + accept/review. See docs/MCP.md for the one-line client config.

Schemas are plain JSON Schema, with each leaf optionally declaring how it is scored (the executable-schema pattern):

{
  "type": "object",
  "properties": {
    "invoice_id": {"type": "string"},
    "vendor":     {"type": "string", "x-scoring": "semantic"},
    "total":      {"type": "number", "x-numeric-tol": 0.01}
  }
}

What's inside

Every metric implements the exact definition in PROJECT.md §5 with a hand-computed numeric regression test (201 tests, eval/ coverage 98%).

Results on real documents

Two independent real OCR extractors (RapidOCR and PaddleOCR) on real CORD receipts and FUNSD forms, scored by the harness (full tables + reading):

• Grounding is a real trust signal: grounded fields are 84–85% correct vs ~1% for ungrounded (gap ≈ +0.84; box accuracy @IoU 0.5 = 0.75–0.78).
• The abstention layer is honest: with a weak field-extractor the base error rate is high, so conformal abstention at a 2–5% budget correctly refuses to auto-accept — you report selective risk, not headline accuracy.
• The synthetic slice (strong extractor) shows the other end: Coverage@2% ≈ 1.0.

The thesis holds on real data: grounding + a learned fusion rank errors; self-reported and single-sample-consensus confidence do not.

The benchmark

make results     # regenerates every table/figure in paper/generated from configs/

The harness runs signals × calibrators × the full metric suite with a document-level calibration split (disjointness asserted in code), bootstrap CIs, and a conformal-guarantee row. It ships a deterministic synthetic slice (runs in CI) plus CORD and FUNSD loaders with gold source boxes; extractor: dispatches to any adapter (rapidocr, paddleocr-vl, …) and dataset: to any slice. See the GPU runbook to reproduce the real-model rows. Core claims (grounding beats verbalized; conformal holds its guarantee) are also CI-enforced as unit tests, not just stated.

Why not just use the parser's own score?

Because it doesn't exist (Docling/MinerU/Marker), or it's a raw recognition score that was never calibrated against field-level correctness (PaddleOCR/dots.ocr). See docs/USP.md for the audit, and the reliability diagrams in paper/generated/ for what "calibrated" actually buys you.

Roadmap

• v0.1 — library + CLI + harness + synthetic benchmark slice + UI
• v0.2 — CORD + FUNSD real slices with gold boxes; learned combiner; 1000× faster grounder
• v0.3 — real-model results (RapidOCR + PaddleOCR on CORD/FUNSD)
• v0.4 — vendor-neutral API-VLM extractor (OpenAI/Anthropic) with k-sample consensus; compilable paper with auto-generated tables
• v0.5 — novel method (grounding-conditioned conformal, +0.50 coverage at fixed risk) + MCP server (agent trust layer) + real frontier-VLM results
• v0.6 — method validated on real data at scale (FUNSD 24%→71% coverage at 2% risk); inter-annotator-agreement tooling (verifydoc iaa); numeric-aware grounding
• dots.ocr via vllm; SROIE / DocILE / XFUND slices; human-labeled correctness at scale
• Paper submission (contributions welcome)

Documentation

• How it works — the pipeline, the abstention idea, why grounding is a trust signal
• Related work & positioning — how VerifyDoc compares to Beyond Logprobs, Cleanlab TLM, conformal factuality, and the commercial IDP stack
• Real-model results — RapidOCR + PaddleOCR numbers and reading
• GPU runbook — reproduce the real-extractor rows
• USP audit · Paper draft · Full spec

Development

git clone https://github.com/bhaskargurram-ai/verifydoc && cd verifydoc
uv venv .venv && uv pip install -e ".[dev]"
make test lint typecheck     # all green before any PR (CI enforces)
make results                 # regenerate benchmark tables + LaTeX
make paper                   # compile the paper (needs a LaTeX toolchain)

Contributions welcome — see the issues tagged good-first-issue. All model-specific code goes in verifydoc/adapters/; a new extractor is one file.

Citation

@software{verifydoc2026,
  author = {Gurram, Bhaskar},
  title  = {VerifyDoc: Calibrated, Abstaining, Grounded Document Extraction},
  year   = {2026},
  url    = {https://github.com/bhaskargurram-ai/verifydoc}
}

Apache-2.0.