ocr-extraction-understanding-eval 0.1.0

Model-agnostic evaluation for user-defined operations over PDF and image data.

OCR Extraction & Understanding Evaluation (visual_evals)

An evaluation pipeline for OCR extraction and downstream manipulation of document-derived data.

Abstract

Systems that read documents rarely stop at optical character recognition. A production pipeline typically extracts structured values from a PDF or image, then manipulates them through one or more downstream operations— recomputing totals, classifying instruments, comparing against a base table, summarizing, or routing. Errors compound across these stages, and a failure observed at the end of the pipeline is not necessarily caused by the stage where it surfaces.

visual_evals is a research-oriented, model-agnostic harness for evaluating such multi-stage systems end to end. It follows the useful part of the Ragas pattern—test cases, pluggable metrics, and inspectable reports—but makes three distinctions that matter for document pipelines:

1. Three levels of reference authority. Every reference carries one of three authority levels, and the harness treats them differently:

   • evidence — a raw source (PDF, image, OCR/Markdown) that can support or contradict a prediction but is not assumed infallible;
   • generated — a provisional, AI-generated ground truth synthesized from the evidence when no human reference exists (see Generating provisional ground truth); agreement among models is a reliability signal, not proof of accuracy;
   • ground_truth — an authoritative human or trusted-system reference.

   A generated reference is never silently treated as authoritative: it is promoted to ground_truth only through an explicit, signed human approval step. Any mixture of the three can appear in a single test case.

2. Deterministic where possible, judged where necessary. Exact metrics run wherever structured ground truth exists; a visual LLM judge handles images, PDFs, layout, semantic equivalence, and rubric-based criteria.

3. Dependency-aware failure attribution. Each stage records its upstream dependencies, so the harness can localize where a pipeline failed rather than only reporting that it failed.

The library evaluates recorded inputs and outputs; it does not dictate or execute the system under test. Domain policy—what each operation is and how its correctness is decided—stays in the calling application.

Research design goals

Goal	Mechanism
Attribute failures across compounding stages	depends_on graph + failure_context
Avoid treating lossy OCR/Markdown as truth	evidence/ground-truth authority levels
Reduce single-judge bias	model-balanced JudgePanelMetric (PoLL-style)
Keep judgments auditable	versioned specs, SHA-256 prompt fingerprints, retained dissent
Separate the prompt-under-test from the correctness rule	OperationSpec vs. independently reviewed EvaluationSpec

The voting and panel design draw on published findings on self-consistency, panels of diverse evaluators, and documented LLM-judge biases; see Research-backed multi-judge voting for citations and the important caveats on what agreement does and does not prove.

Core contract

                         ┌─ PDF / image ─────────── evidence
prediction JSON ─┬───────┼─ OCR / Markdown ─────── evidence
                 │       ├─ human notes ────────── ground_truth
                 │       └─ JSON / CSV base table ground_truth
                 ▼
       deterministic metrics + visual LLM judge
                 ▼
       JSON report + Markdown summary

• Evidence can support or contradict a prediction, but is not assumed infallible.
• Ground truth is an authoritative human or trusted-system reference.
• References are interchangeable and can be mixed in one test case.
• Exact metrics run where structured ground truth exists.
• The visual judge handles images, PDFs, layouts, semantic equivalence and rubrics.

The extraction → manipulation pipeline

Evaluation does not stop after OCR or extraction. The unit of study is the whole chain from source document to final decision:

document ──▶ OCR / extraction ──▶ manipulation A ──▶ manipulation B ──▶ ... ──▶ decision
                                          │                 │                        │
                                  chosen metric      chosen metric           chosen metric

An operation can be extraction, calculation, comparison, validation, summarization, routing, or anything else the caller defines (LLM prompt). operation is a free-form string, not an enum or a library-controlled taxonomy. The library evaluates recorded inputs and outputs; it does not dictate or execute the system being tested.

Each EvaluationCase represents one stage and contains:

• action_instruction: the exact editable prompt or rules given to the action;
• input_data: the material received from an earlier stage;
• prediction: this stage's output;
• references: any mixture of evidence and authoritative ground truth;
• rubric: the caller's success rule for this operation;
• judge_instruction: optional stage-specific guidance for the evaluator;
• operation: a free-form identifier used to select metrics; and
• depends_on: upstream stage IDs.

PipelineEvaluator selects metrics in this order: exact stage ID, operation, then default metrics. An operation can use a deterministic built-in metric, an arbitrary Python function through FunctionMetric, an LLM judge, or several of these together. This keeps domain policy in the application using the library.

The caller defines both sides of the contract:

1. operation and rubric say what the system was supposed to do.
2. input_data and prediction record what went in and what came out.
3. A metric says how correctness is decided. Use code when the rule is exact (such as arithmetic), and an LLM judge when the rule needs semantic or visual judgment.

Dependency-aware failure diagnosis

Every stage report includes its own verdict, the verdicts of all upstream stages, and a failure_context:

• current_stage: this stage failed while its upstream stages passed;
• upstream: this stage passed on the input it received, but an upstream stage failed;
• upstream_or_combined: both this stage and an upstream stage failed, so the result must not be blamed on this stage alone;
• indeterminate: an uncertain or non-applicable result prevents attribution.

This is careful failure localization, not automatic proof of causality. If both extraction and a later action fail, rerun the action with corrected extraction input. If it then passes, the extraction error caused the downstream failure; if it still fails, the action also has an independent problem.

Stage checkpointing

Long multi-model evaluations can checkpoint after every completed stage:

report = evaluator.evaluate(
    pipeline,
    checkpoint_dir="eval_runs/my-case/checkpoints",
    checkpoint_context={
        "judge_provider": "anthropic",
        "judge_model": "claude-opus-4-8",
    },
)

Each report is written atomically. A rerun loads completed stages instead of calling their metrics or judges again. A checkpoint is reused only when the serialized stage case and checkpoint_context have the same fingerprint, so include every external setting that can change the result, especially judge provider, model, temperature, and prompt version.

Installation

The project is a standard src-layout Python package built with Hatchling. Install it into your environment as an editable dependency:

pip install -e .

The import package is visual_evals; the distribution is ocr-extraction-understanding-eval:

import visual_evals

The package does not create, load or require a .env file. Secret configuration belongs to the application using the library.

For development and to run the test suite:

pip install -e ".[dev]"
pytest -q

Optional extras:

• pip install -e ".[pdf]" — PyMuPDF4LLM Markdown conversion for search/debug.

Building and distributing

Because packaging metadata lives entirely in pyproject.toml, a wheel and source distribution can be produced with the standard build frontend:

pip install build
python -m build          # writes dist/*.whl and dist/*.tar.gz

The resulting wheel is installable anywhere with pip install dist/*.whl, and publishable to a package index with twine upload dist/* once you own the name.

Direct PDF evaluation quick start

Keep the original PDF as primary evidence. The system output is compared directly with the PDF; it is not evaluated against a lossy Markdown replacement:

from visual_evals import (
    EvaluationCase,
    Evaluator,
    LLMJudgeMetric,
    ModelJudge,
    OpenAIResponsesAdapter,
    artifact_from_path,
)

source = artifact_from_path("statement.pdf", authority="evidence")
case = EvaluationCase(
    case_id="statement-001",
    operation="statement_extraction",
    prediction=actual_system_output,
    # Optional. Omit this entirely if the producer is unknown.
    producer={
        "provider": "anthropic",
        "model": "claude-sonnet-your-version",
    },
    references=[source],
    rubric=(
        "Every extracted value must match the visible PDF and remain associated "
        "with the correct label, row, column, and page."
    ),
)
adapter = OpenAIResponsesAdapter(model="gpt-4o-mini")
report = Evaluator(
    [LLMJudgeMetric(ModelJudge(adapter))]
).evaluate(case)
print(report.verdict)
print(adapter.total_usage.model_dump())

Producer provenance is optional. If omitted, reports show unknown and make no assumption about judge independence. If supplied, judges are labelled same_model, same_provider, or independent. Producer-related votes remain visible, but a panel with no independent judge requires human review.

Set OPENAI_API_KEY and ANTHROPIC_API_KEY in the process that runs this code. The library never parses .env; .env is git-ignored only to reduce the risk of accidentally committing a locally created secret file.

The runnable two-model example evaluates a JSON prediction directly against the PDF:

python examples/evaluate_pdf_directly.py statement.pdf prediction.json

Markdown conversion remains available as an optional local derivative via PyMuPDF4LLM. It is useful for search and debugging but never replaces the raw PDF:

pip install -e ".[pdf]"
python examples/pdf_to_markdown.py statement.pdf

See the full usage guide for provider selection, voting, ground truth, single-vs-panel batch reports, token semantics, and Windows commands.

Python API

This example evaluates extraction and a subsequent calculation. The custom metric independently recomputes the total from the stage input:

from visual_evals import (
    EvaluationCase,
    ExactStructuredMetric,
    FunctionMetric,
    PipelineCase,
    PipelineEvaluator,
    artifact_from_path,
    artifact_from_value,
)

extraction = EvaluationCase(
    case_id="document-001",
    stage_id="extract",
    operation="parse_line_items",
    prediction={
        "line_items": [
            {"quantity": 2, "unit_price": 4.25},
            {"quantity": 1, "unit_price": 3.00},
        ]
    },
    references=[
        artifact_from_path("invoice.png", authority="evidence"),
        artifact_from_value(
            {
                "line_items": [
                    {"quantity": 2, "unit_price": 4.25},
                    {"quantity": 1, "unit_price": 3.00},
                ]
            },
            authority="ground_truth",
            artifact_id="human-transcription",
        ),
    ],
)

calculation = EvaluationCase(
    case_id="document-001",
    stage_id="calculate-total",
    operation="sum_quantity_times_unit_price",
    input_data=extraction.prediction,
    prediction={"total": 11.50},
    depends_on=["extract"],
    rubric="Total equals the sum of quantity multiplied by unit price.",
)

def total_matches(stage):
    expected = sum(
        row["quantity"] * row["unit_price"]
        for row in stage.input_data["line_items"]
    )
    return stage.prediction["total"] == expected

pipeline = PipelineCase(
    pipeline_id="document-processing",
    stages=[extraction, calculation],
)
report = PipelineEvaluator(
    operation_metrics={
        "parse_line_items": [ExactStructuredMetric()],
        "sum_quantity_times_unit_price": [
            FunctionMetric(
                "recalculate_total",
                total_matches,
                score_name="calculation_accuracy",
            )
        ],
    }
).evaluate(pipeline)

Custom action and judge instructions

The downstream action prompt is part of the system under test, not a source of truth. Define it once in an OperationSpec, alongside a separate, independently reviewed EvaluationSpec. Every case that names the operation reuses that configuration:

from visual_evals import (
    EvaluationCase,
    EvaluationSpec,
    OperationRegistry,
    OperationSpec,
    build_judge_request,
)

vehicle_operation = OperationSpec(
    spec_id="investment-vehicle",
    operation="vehicle_classification",
    version="2026-07-23",
    # This is the real production prompt being tested. It may be imperfect.
    action_instruction="Determine the investment vehicle.",
    # These rules are reviewed independently; they define correctness.
    evaluation_spec=EvaluationSpec(
        spec_id="investment-vehicle-correctness",
        version="3",
        status="approved",
        objective="Choose the vehicle using the approved investment taxonomy.",
        criteria=[
            "Use fund for pooled capital managed as one portfolio.",
            "Use equity only for direct ownership in a company.",
            "Use unknown when the evidence is insufficient.",
        ],
        allowed_outputs=["fund", "equity", "bond", "unknown"],
    ),
)

vehicle_stage = EvaluationCase(
    case_id="statement-001",
    stage_id="classify-vehicle",
    operation="vehicle_classification",
    input_data=extraction.prediction,
    prediction={"vehicle": "fund"},
    depends_on=["extract"],
)

# Evaluator and PipelineEvaluator accept operation_specs=[vehicle_operation].
# Resolve manually only when you want to inspect the prompt before evaluation.
resolved_stage = OperationRegistry([vehicle_operation]).resolve(vehicle_stage)

# Inspect exactly what any judge provider will receive before calling a model.
request = build_judge_request(resolved_stage)
print(request.task["action_instruction"])
print(request.task["evaluation_spec"])
print(request.instructions)

action_instruction records what the action model was asked to perform; the library never treats it as truth and does not execute the action. EvaluationSpec.status="approved" records that the correctness rules were reviewed; draft specifications must still be calibrated.

judge_instruction can add stage-specific evaluation guidance. Its default mode is append, which keeps the protected evidence and response-contract rules. Replacing that base prompt requires both judge_instruction_mode="replace" and allow_unsafe_judge_override=True; reports display a warning until the replacement is calibrated against human-labelled cases.

Reports include the complete instructions and evaluation criteria, their versions, and separate SHA-256 fingerprints. This makes prompt changes visible without asking users to paste the same configuration into every test case.

Model-agnostic judges

The scoring layer depends on the Judge protocol, not on an OpenAI class. Every provider receives the same JudgeRequest:

• instructions: the shared evaluator prompt;
• task: operation, input, prediction claims, rubric, and reference inventory;
• output_schema: the JSON Schema for JudgeOutput; and
• references: text, structured data, image, PDF, or file artifacts.

There is one ModelJudge. OpenAI, Claude, and local models differ only in the adapter passed to it. OpenAI and Anthropic have ready-to-use transport adapters:

from visual_evals import (
    AnthropicMessagesAdapter,
    LLMJudgeMetric,
    ModelJudge,
    OpenAIResponsesAdapter,
)

openai_metric = LLMJudgeMetric(
    ModelJudge(OpenAIResponsesAdapter(api_key=application_secret))
)
anthropic_metric = LLMJudgeMetric(
    ModelJudge(AnthropicMessagesAdapter(api_key=anthropic_secret))
)

Any Ollama, vLLM, Transformers, or other model client can be wrapped by the same judge:

from visual_evals import LLMJudgeMetric, ModelJudge

def call_model(request):
    # This adapter owns the provider-specific SDK or local inference call.
    # Attach request.references when the model supports images or PDFs.
    result = my_model_adapter.generate_json(
        instructions=request.instructions,
        task=request.task,
        schema=request.output_schema,
        references=request.references,
    )
    return result  # JudgeOutput, a matching dict, or matching JSON text

metric = LLMJudgeMetric(
    ModelJudge(call_model),
    name="claude-sonnet-judge",  # or "ollama-judge", "vllm-judge", etc.
)

The evaluation cases, evidence semantics, dependency diagnosis, metric calculation, and reports are identical across providers. Only call_model changes. A stage can also use deterministic metrics and multiple judges together. Give each judge metric a distinct name when comparing models in one report.

If api_key is omitted, the built-in adapters read OPENAI_API_KEY or ANTHROPIC_API_KEY from the caller's existing process environment. The library never reads an environment file. Both adapters expose last_usage, usage_records, total_usage, and reset_usage().

Research-backed multi-judge voting

JudgePanelMetric supports both different model providers and repeated samples at different temperatures:

from visual_evals import (
    JudgePanelMetric,
    JudgeVariant,
    ModelJudge,
    OpenAIResponsesAdapter,
)

panel = JudgePanelMetric(
    [
        JudgeVariant(
            name="openai-low-temp",
            judge=ModelJudge(
                OpenAIResponsesAdapter(
                    model=openai_model_id,
                    temperature=0.0,
                    api_key=openai_secret,
                )
            ),
            provider="openai",
            model=openai_model_id,
            temperature=0.0,
        ),
        JudgeVariant(
            name="openai-higher-temp",
            judge=ModelJudge(
                OpenAIResponsesAdapter(
                    model=openai_model_id,
                    temperature=0.8,
                    api_key=openai_secret,
                )
            ),
            provider="openai",
            model=openai_model_id,
            temperature=0.8,
        ),
        JudgeVariant(
            name="claude",
            judge=ModelJudge(call_claude_at_temperature_0),
            provider="anthropic",
            model=claude_model_id,
            temperature=0.0,
        ),
        JudgeVariant(
            name="local",
            judge=ModelJudge(call_local_model),
            provider="local",
            model=local_model_id,
            temperature=0.2,
        ),
    ],
    vote_rule="majority",             # or "unanimous"
    review_on_any_disagreement=True,  # retain dissent for human review
)

Voting is model-balanced:

1. Repeated runs or temperature variants of the same provider/model form one self-consistency group.
2. Each distinct provider/model group contributes one final vote.
3. A majority or unanimous rule selects the panel verdict.
4. Ties, uncertainty, and—by default—any minority dissent enter the human-review queue.

This prevents five temperature samples from one model outvoting one Claude and one local-model judge. Reports preserve raw-sample agreement, model-group agreement, every variant's provider/model/temperature, the winning vote, and dissenting evidence.

The design follows two related but different research findings:

• Wang et al.'s ICLR self-consistency work found that sampling multiple reasoning paths and selecting the consistent answer improved several reasoning benchmarks. This supports repeated stochastic samples, but is not direct proof that temperature voting makes an LLM judge accurate. Google Research paper
• Verga et al.'s Panel of LLM evaluators (PoLL) found that panels of diverse model families outperformed a single large judge across their tested evaluation settings and reduced intra-model bias. PoLL paper
• Zheng et al. documented position, verbosity, self-enhancement, and reasoning limitations in LLM judges, supporting calibration and retained human review. MT-Bench/Chatbot Arena paper

The OpenAI Responses API documents temperature values from 0 to 2 and advises changing temperature or top_p, not both. Official Responses API reference Other providers and model families may expose different sampling controls.

Voting agreement remains a reliability signal, not accuracy. Validate the panel against human-labelled cases before using it for production gating.

Generating provisional ground truth when none exists

When the only material you have is an image or a set of pages—and no human has transcribed them—you do not have ground truth yet. You have evidence. This section describes how to generate a provisional, clearly-labelled authority="generated" reference from that evidence, and how to promote it to authoritative ground_truth only through explicit human approval.

How generation works, end to end:

1. Supply the evidence and the extraction contract. Give GroundTruthPanel two or more independent model providers, an extraction_instruction, and the expected_fields you want read out.
2. Each provider reads the source blind. Every provider sees only the source images and the extraction instruction—never the system prediction and never another model's output. Each independently emits candidate claims (path, expected_value, cited reference_id, visual evidence, and a confidence).
3. Claims are reconciled field by field. For each field the panel takes a normalized vote across providers. Fields that meet minimum_agreement become consensus rows; anything below it—including a field one model omitted entirely—becomes a disagreement and sets requires_human_review=True.
4. A provisional reference is assembled. The consensus rows are packaged into a single ReferenceArtifact with authority="generated". It is safe to evaluate against immediately, but any score is agreement with a provisional AI reference, not verified accuracy.
5. (Optional) reduce the review queue automatically. Run GroundTruthAdjudicator for a blind, targeted second read of only the disputed paths, or resolve_ground_truth_union(...) to accept non-conflicting one-sided claims. Both keep authority="generated".
6. Promote to ground truth via signed human approval. A generated reference becomes authoritative only through approve_generated_reference(...) (or the file-based write_ground_truth_review → approved_reference_from_review flow), which records approved_by and refuses to approve while any disagreement is unresolved.

Terminology: what the panel produces is generated ground truth (authority="generated"), not evidence. Evidence is the raw source you feed in; the generated reference is a hypothesis about what that evidence says. Only human approval turns it into authority="ground_truth".

Each provider sees only the source images and extraction instructions—not the system prediction—and independently generates candidate reference facts:

from visual_evals import (
    EvaluationCase,
    Evaluator,
    ExactStructuredMetric,
    GroundTruthPanel,
    GroundTruthRouter,
    OpenAIResponsesAdapter,
    approve_generated_reference,
    artifact_from_path,
)

source_images = [
    artifact_from_path(
        "page-001.png",
        authority="evidence",
        artifact_id="page-001",
        metadata={"page": 1},
    ),
    artifact_from_path(
        "page-002.png",
        authority="evidence",
        artifact_id="page-002",
        metadata={"page": 2},
    ),
]

panel = GroundTruthPanel(
    {
        "openai": OpenAIResponsesAdapter(api_key=openai_secret),
        "claude": call_claude,
        "local-model": call_local_model,
    },
    extraction_instruction="Extract currency, amount, and account name.",
    expected_fields=["$.currency", "$.amount", "$.account_name"],
    minimum_agreement=1.0,  # Any disagreement creates a review item.
)

router = GroundTruthRouter(image_panel=panel)

prepared = router.prepare(
    case_id="statement-001",
    references=source_images,
)
generation = prepared.image_report

# Agreed claims can be used provisionally, but are not called ground truth.
case = EvaluationCase(
    case_id="statement-001",
    operation="statement_extraction",
    prediction=actual_extraction,
    references=[*source_images, generation.provisional_reference],
)
report = Evaluator([ExactStructuredMetric()]).evaluate(case)

# If providers disagreed, a human supplies the resolved path/value rows.
approved_reference = approve_generated_reference(
    generation.provisional_reference,
    approved_by="[e-mail]",
    resolved_content=human_resolved_rows,
)

Routing is based on the complete input set:

all PDF/ODF/ODT documents -> preserve original source evidence
all images                -> preserve sources + optional model consensus
explicit markdown mode    -> add a local supplementary derivative
mixed/unsupported inputs  -> explicit error; prepare separately

GroundTruthRouter uses document_mode="preserve" by default. Optional Markdown derivatives have authority="generated" and never replace the raw document. They are unscored or low-confidence until evaluated and therefore remain reviewable.

The generation report contains:

• every provider's independently generated candidate claims;
• fields on which the models agreed;
• each disputed field and candidate value;
• expected fields omitted by every provider;
• cited image IDs, page metadata, and visual evidence; and
• requires_human_review=True whenever candidates differ or no model produced a usable claim.

The consensus artifact has authority="generated". Evaluation against it is labelled agreement with a provisional AI-generated reference, not accuracy. It can be promoted to authority="ground_truth" only through approve_generated_reference; unresolved disagreements require human-supplied resolved content. Even unanimous models can share errors, so human calibration is still recommended.

For a persistent review workflow, use write_ground_truth_review before the tested pipeline runs. A reviewer edits the generated claims, resolves each review_items entry with decision="set" or "omit", then sets status="approved" and approved_by. Load the result with approved_reference_from_review; it rejects unresolved or unsigned files. Freeze ground truth for the complete batch before generating predictions.

GroundTruthAdjudicator can reduce the manual queue with a second blind source-reading pass. It shows each provider only the disputed paths, never the first-pass candidate values. The default requires unanimity; callers can set minimum_agreement for a majority-vote panel. Votes below the configured threshold remain human-review items.

For unattended experiments where omission by one generator should not remove a field found by another, call resolve_ground_truth_union(generation). It accepts non-conflicting present values, leaves conflicting present values unresolved, and keeps the reference authority as generated. Its scores remain provisional agreement signals rather than verified accuracy.

CLI

The CLI uses the caller's existing OPENAI_API_KEY or ANTHROPIC_API_KEY process environment. With --judge auto, OpenAI is preferred when both exist, then Anthropic; without either key, only deterministic metrics run. A model can be supplied through --model or VISUAL_EVAL_MODEL, and OpenAI reasoning effort through --reasoning-effort or VISUAL_EVAL_REASONING_EFFORT.

Evaluate directly against the statement:

python -m visual_evals.cli \
  --prediction examples/output.json \
  --evidence examples/source.pdf \
  --output reports/example-visual-eval.json \
  --markdown reports/example-visual-eval.md

Add a human-maintained base table:

python -m visual_evals.cli \
  --prediction examples/output.json \
  --evidence examples/source.pdf \
  --ground-truth examples/truth.csv \
  --output reports/example-grounded-eval.json

Use an image and human notes instead:

python -m visual_evals.cli \
  --prediction output.json \
  --evidence statement-page.jpg \
  --ground-truth main-points.md \
  --output reports/image-eval.json

Run deterministic checks only:

python -m visual_evals.cli \
  --prediction output.json \
  --ground-truth truth.csv \
  --judge none \
  --output reports/exact.json

Ground truth can use any format

Ground truth means “authoritative reference”; it does not require a special table shape. You can supply:

• an ordinary CSV/TSV table with any columns;
• free-form .md or .txt review notes;
• JSON;
• a PDF, image, or document supported by the selected model adapter; or
• several references in different formats.

The LLM judge reads the reference and compares its meaning with the extraction. This supports cases where a reviewer provides only the important facts:

topic,main_point
vehicle,Pooled investment fund
risk,Moderate risk
currency,USD

That semantic comparison is probabilistic. It should be reported as an LLM judgment against authoritative material, not deterministic field accuracy.

For deterministic partial field metrics, optionally use explicit paths:

path,expected
$.heading,Safety inspection
$.status_text,Needs review
$.items[0].code,A-12
$.items[0].quantity,3

field_path is accepted instead of path; expected_value or value is accepted instead of expected.

Reference comparison modes are:

• auto: explicit path/value tables and JSON can use exact metrics; ordinary tables and notes use semantic comparison;
• semantic: force LLM comparison even for structured JSON; and
• exact: explicitly flatten structured data for deterministic comparison.

reference = artifact_from_path(
    "review_notes.csv",
    authority="ground_truth",
    comparison_mode="semantic",
)

Exact metrics skip references that are not exact-compatible rather than producing misleading missing-field scores.

Interpreting scores

• field_accuracy: correct authoritative fields / all annotated fields.
• claim_precision: supported predictions / judge-decided predictions.
• reference_recall: matched reference claims / judge-decided reference claims.
• claim_verification_coverage: fraction of claims the judge could decide.
• f1: harmonic mean of claim precision and reference recall.

With source evidence only, recall is an LLM-estimated inventory—not true accuracy. With arbitrary notes or tables marked as ground truth, the reference is authoritative but the LLM's interpretation remains probabilistic. Calibrate the judge on human labels and report judge/human agreement before scaling to a large test set.