VerifyDoc

The trust layer for document → structured-JSON extraction. Wrap any extractor — get back JSON where every field carries a calibrated confidence, a source grounding (page + bbox / char span), and an accept/review decision tuned to your error budget.

Above: a real pipeline run (scripts/make_demo_gif.py). The extractor returned $1,432.50; the page says $1,234.50. Grounding support drops to 0.78, the field misses the accept threshold, and the reviewer is pointed at the exact source region. The other three fields are auto-accepted.

The problem

Modern document parsers read pages at 96%+ benchmark accuracy — and still emit fluent, plausible, silently-wrong values ($42.50 → $45.20) with no reliable per-field signal telling you which values to trust. Commercial APIs (Box, Azure, Textract) sell field-level confidence as a closed feature. No popular open-source parser leads with it. (full USP audit)

VerifyDoc doesn't compete with the parsers — it layers on top of any of them:

document + schema ─► ingest ─► extractor adapter ─► confidence ─► calibration
                                (any model)          signals       (fit on cal split)
                     ─► grounding ─► abstention policy ─► verified JSON + review UI
                        (bbox/span)   (target risk α)

At a chosen operating point, VerifyDoc auto-accepts as many fields as possible while holding the error rate among accepted fields below your target (e.g. ≤ 2%) — everything else is routed to review with its source location attached, so a human verifies in seconds instead of eyeballing every field.

Quickstart

pip install verifydoc          # core (text pipelines + eval harness)
pip install 'verifydoc[pdf]'   # + PDF/image ingestion

from verifydoc import verify

# ready-to-run sample lives in examples/
result = verify("examples/invoice.txt", schema="examples/invoice_schema.json")
for f in result.fields:
    print(f"{f.path:12} = {f.value!r:24} conf={f.confidence:.2f} {f.decision}")
    if f.grounding:
        print(f"             └─ page {f.grounding.page}, bbox {f.grounding.bbox}")

verifydoc extract examples/invoice.txt --schema examples/invoice_schema.json --threshold 0.8
streamlit run ui/streamlit_app.py     # review UI: green/red fields + click-through to source

See examples/ for the full runnable walk-through.

For AI agents (MCP)

Give any MCP-capable agent (Claude Desktop, IDEs, custom agents) a trust layer for reading documents — so it acts on confident, grounded fields and escalates the rest instead of hallucinating forward:

pip install 'verifydoc[mcp]'
verifydoc-mcp     # stdio MCP server exposing verify_extraction()

Every field the agent extracts comes back with confidence + grounding + accept/review. See docs/MCP.md for the one-line client config.

Schemas are plain JSON Schema, with each leaf optionally declaring how it is scored (the executable-schema pattern):

{
  "type": "object",
  "properties": {
    "invoice_id": {"type": "string"},
    "vendor":     {"type": "string", "x-scoring": "semantic"},
    "total":      {"type": "number", "x-numeric-tol": 0.01}
  }
}

What's inside

Every metric implements the exact definition in PROJECT.md §5 with a hand-computed numeric regression test (201 tests, eval/ coverage 98%).

Results on real documents

Two independent real OCR extractors (RapidOCR and PaddleOCR) on real CORD receipts and FUNSD forms, scored by the harness (regenerate the full tables locally with make results):

• Grounding is a real trust signal: grounded fields are 84–85% correct vs ~1% for ungrounded (gap ≈ +0.84; box accuracy @IoU 0.5 = 0.75–0.78).
• The abstention layer is honest: with a weak field-extractor the base error rate is high, so conformal abstention at a 2–5% budget correctly refuses to auto-accept — you report selective risk, not headline accuracy.
• The synthetic slice (strong extractor) shows the other end: Coverage@2% ≈ 1.0.

The thesis holds on real data: grounding + a learned fusion rank errors; self-reported and single-sample-consensus confidence do not.

The benchmark

make results     # regenerates every benchmark table/figure from configs/ (local output)

The harness runs signals × calibrators × the full metric suite with a document-level calibration split (disjointness asserted in code), bootstrap CIs, and a conformal-guarantee row. It ships a deterministic synthetic slice (runs in CI) plus CORD and FUNSD loaders with gold source boxes; extractor: dispatches to any adapter (rapidocr, paddleocr-vl, …) and dataset: to any slice. See the GPU runbook to reproduce the real-model rows. Core claims (grounding beats verbalized; conformal holds its guarantee) are also CI-enforced as unit tests, not just stated.

Why not just use the parser's own score?

Because it doesn't exist (Docling/MinerU/Marker), or it's a raw recognition score that was never calibrated against field-level correctness (PaddleOCR/dots.ocr). See docs/USP.md for the audit, and the reliability diagrams make results produces for what "calibrated" actually buys you.

Roadmap

• v0.1 — library + CLI + harness + synthetic benchmark slice + UI
• v0.2 — CORD + FUNSD real slices with gold boxes; learned combiner; 1000× faster grounder
• v0.3 — real-model results (RapidOCR + PaddleOCR on CORD/FUNSD)
• v0.4 — vendor-neutral API-VLM extractor (OpenAI/Anthropic) with k-sample consensus; compilable paper with auto-generated tables
• v0.5 — novel method (grounding-conditioned conformal, +0.50 coverage at fixed risk) + MCP server (agent trust layer) + real frontier-VLM results
• v0.6 — method validated on real data at scale (FUNSD 24%→71% coverage at 2% risk); inter-annotator-agreement tooling (verifydoc iaa); numeric-aware grounding
• dots.ocr via vllm; SROIE / DocILE / XFUND slices; human-labeled correctness at scale
• Paper submission (contributions welcome)

Documentation

• Framework integrations — drop-in trust layer for Instructor, Pydantic-AI, Outlines, LangChain
• How it works — the pipeline, the abstention idea, why grounding is a trust signal
• Related work & positioning — how VerifyDoc compares to Beyond Logprobs, Cleanlab TLM, conformal factuality, and the commercial IDP stack
• Framework integrations — Instructor / Pydantic / LangChain drop-in
• GPU runbook — reproduce the real-extractor rows
• USP audit · Full spec

Development

git clone https://github.com/bhaskargurram-ai/verifydoc && cd verifydoc
uv venv .venv && uv pip install -e ".[dev]"
make test lint typecheck     # all green before any PR (CI enforces)
make results                 # regenerate benchmark tables + LaTeX
make paper                   # compile the paper (needs a LaTeX toolchain)

Contributions welcome — see the issues tagged good-first-issue. All model-specific code goes in verifydoc/adapters/; a new extractor is one file.

Citation

@software{verifydoc2026,
  author = {Gurram, Bhaskar},
  title  = {VerifyDoc: Calibrated, Abstaining, Grounded Document Extraction},
  year   = {2026},
  url    = {https://github.com/bhaskargurram-ai/verifydoc}
}

Apache-2.0.