LLM Validation Framework

A comprehensive Python framework for evaluating LLM-extracted structured data against ground truth labels. Supports binary classification, scalar values, and list fields with detailed performance metrics, confidence-based evaluation, and statistical uncertainty quantification via non-parametric bootstrap confidence intervals.

📄 Paper

The methodology behind this framework is described in a medRxiv preprint: medRxiv 2026.05.18.26353541 — DOI: 10.64898/2026.05.18.26353541

✨ Key Features

• Multi-field validation - Binary (True/False), scalar (single values), and list (multiple values) data types
• Coded field support - Score a value together with its code (ICD-10 / ICD-O-3, RxNorm) as two independent facets
• Partial labeling support - Handle datasets where different cases have labels for different subsets of fields
• Parent matching - Hierarchical dictionaries: a prediction that is the parent of the labeled value scores a partial match (0.5) instead of incorrect. E.g. for ICD-10: C00 "Malignant neoplasm of lip" is partially correct for label C00.1 "Malignant neoplasm, external lower lip".
• Dual usage modes - Validate pre-computed results OR run live LLM inference with validation
• Comprehensive metrics - Precision, recall, F1/F2, accuracy, specificity, with micro and macro aggregation where applicable
• Confidence analysis - Automatic performance breakdown by confidence levels
• Statistical uncertainty - Non-parametric bootstrap confidence intervals for all performance metrics
• Production ready - Parallel processing, intelligent caching, detailed progress tracking

🚀 Quick Start

Prerequisites

# Install from PyPI
pip install llmvalidate

# OR install from source
pip install -r requirements.txt  # Python 3.11+ required

Demo

python runme.py

Processes the included samples.csv (14 test cases covering all validation scenarios) and outputs timestamped results to validation_results/samples/:

• Results CSV - Row-by-row comparison with confusion matrix counts and item-level details
• Metrics CSV - Aggregated performance statistics with confidence breakdowns
• CI Metrics CSV - Confidence intervals for metrics

📊 Usage Modes

Mode 1: Validate Existing Results

When you have LLM predictions in Res: {Field Name} columns:

import pandas as pd
from llmvalidate import validate

df = pd.read_csv("data.csv", index_col="Patient ID")
# df must contain: "Field Name" and "Res: Field Name" columns

results_df, metrics_df = validate(
    source_df=df,
    fields=["Diagnosis", "Treatment"],  # or None for auto-detection
    structure_callback=None,
    output_folder="validation_results"
)

Mode 2: Live LLM Inference + Validation

from llmvalidate.structured import StructuredResult, StructuredGroup, StructuredField
from llmvalidate.utils import flatten_structured_result

def llm_callback(row, i, raw_text_column_name):
    raw_text = row[raw_text_column_name]
    # Your LLM inference logic here
    result = StructuredResult(
        groups=[StructuredGroup(
            group_name="medical",
            fields=[
                StructuredField(name="Diagnosis", value="Cancer", confidence="High"),
                StructuredField(name="Treatment", value=["Drug A"], confidence="Medium")
            ]
        )]
    )
    return flatten_structured_result(result), {}

results_df, metrics_df = validate(
    source_df=df,
    fields=["Diagnosis", "Treatment"],
    structure_callback=llm_callback,
    raw_text_column_name="medical_report",
    output_folder="validation_results",
    max_workers=4
)

📋 Input Data Requirements

DataFrame Format

• Unique index - Each row must have a unique identifier (e.g., "Patient ID")
• Label columns - Ground truth values for each field you want to validate (unless the field is declared in derived_fields — see Derived fields)
• Result columns (Mode 1 only) - LLM predictions as Res: {Field Name} columns
• Raw text column (Mode 2 only) - Source text for LLM inference (e.g., "medical_report")

Supported Field Types

Coded Fields (value + code)

A StructuredField may carry an optional code alongside its value — for coded concepts such as ICD-10 / ICD-O-3 topography-morphology or RxNorm drug codes:

StructuredField(name="Primary Histology", value="Adenocarcinoma", code="8140/3",
                confidence="High")

When code is set, the field flattens to two scored facets — Primary Histology-value and Primary Histology-code — each compared against the label column of the same name. confidence / justification stay attached to the logical field (a single Res: Primary Histology confidence column), and its confidence breakdown is applied to both facets. value and code may be lists (e.g. drug names + their RxNorm codes), scored set-wise per facet.

When code is None (the default) the field flattens to a single <name> column as before — fully backward-compatible. A coded field whose code is genuinely unknown should still pass code="-" (the no-information sentinel), not None, so it keeps aligning with the -value / -code label columns.

Special Value Handling

• "-" = Labeled as "No information is available in the source document"
• null/empty/NaN = Field not labeled/evaluated (supports partial labeling where different cases may have labels for different field subsets)
• Lists - Can be Python lists ["a", "b"] or stringified "['a', 'b']" (auto-converted)

The "-"-vs-empty distinction above applies to label columns only. In prediction (Res:) columns there is no "not evaluated" state: "-", "", null/NaN, whitespace-only strings and [] are all treated identically as "nothing extracted" (scoring Mis against a labeled value, or TN when the label is "-"). We still recommend emitting "-" uniformly, to keep predictions symmetric with labels. Exception — binary fields: predictions must be an explicit True/False; an empty prediction is "not True / not False", so it scores FN against a True label and FP against a False label.

Partial Labeling Support

The framework supports partial labeling scenarios where:

• Not every case needs labels for every field
• Different cases can have labels for different subsets of fields
• Missing labels (null/NaN) are handled gracefully in all metrics calculations
• Use "-" when the document explicitly lacks information about a field
• Use null/NaN when the field simply wasn't labeled for that case

📈 Output Files

A validate() run generates two timestamped CSV files (a third, CI metrics, is added when you also run bootstrap_CI — see the demo above):

1. Results CSV (YYYY-MM-DD HH-MM-SS results.csv)

Row-level analysis with detailed per-case metrics:

Original Data:

• All input columns (labels, raw text, etc.)
• Res: {Field} columns with LLM predictions (coded fields split into Res: {Field}-value and Res: {Field}-code)
• Res: {Field} confidence and Res: {Field} justification (if available; kept per logical field, and shared across a coded field's two facets)

Binary Fields:

• TP/FP/FN/TN: {Field} - Confusion matrix counts (1 or 0 per row)

Non-Binary Fields:

• Cor/Mis/Spu: {Field} - Item counts per row (Inc and TN are None for list fields)
• Cor/Inc/Mis/Spu: {Field} items - Actual item lists
• Par: {Field} / Par: {Field} items - Partial-match count and the labeled items whose parent was predicted (only when the field has a hierarchy entry; see Hierarchical / partial match)
• Precision/Recall/F1/F2: {Field} - Per-row metrics (list fields only)

System Columns:

• Sys: from cache - Whether result was cached (speeds up duplicate text)
• Sys: exception - Error information if processing failed
• Sys: time taken - Processing time per row in seconds

2. Metrics CSV (YYYY-MM-DD HH-MM-SS metrics.csv)

Aggregated statistics with confidence breakdowns:

Core Information:

• field - Field name being evaluated
• confidence - Confidence level ("Overall", "High", "Medium", "Low", etc.)
• labeled cases - Total rows with ground truth labels
• field-present cases - Rows where document has information about the field (label is not '-')

Binary Metrics: TP, TN, FP, FN, precision, recall, F1/F2, accuracy, specificity

Non-Binary Metrics: cor, inc, mis, spu, TN, precision/recall/F1/F2 (micro and macro), specificity

Applicability: cor, mis, spu and the (micro) precision/recall/F-scores apply to all non-binary fields; inc, TN and specificity are meaningful for scalar fields only; the (macro) metrics are averages of the per-row metrics and exist for list fields only. accuracy and specificity carry no (micro)/(macro) tag: they are only ever computed one way (pooled counts), since a per-row version would be a 0/1 indicator whose average equals the pooled value.

⚡ Performance Metrics Explained

Binary Classification Metrics

For fields with True/False values (e.g., "Has metastasis"):

Confusion Matrix Counts

Binary Classification Formulas

Structured Extraction Metrics

For scalar and list fields (e.g., "Diagnosis", "Treatment Drugs"):

Core Counts (Per Case Analysis)

Structured Extraction Formulas

Note: Inc and TN (and therefore specificity) are defined only for scalar fields. For list fields, extracted items are always classified as correct, missing, or spurious — Inc stays empty and no TN column is emitted.

The following formulas apply to both binary classification and structured extraction metrics:

Where P = Precision and R = Recall (calculated differently for each metric type).

Hierarchical / Partial Match

Some concepts are hierarchical: a prediction can be correct but less specific than the label (e.g. label TNBC (Triple Negative Breast Cancer), prediction BC (Breast Cancer)). Pass a per-field hierarchy to validate to credit these as partial matches instead of outright wrong:

results_df, metrics_df = validate(
    source_df=df,
    fields=["Diagnosis", "Stage"],
    structure_callback=None,
    hierarchy={
        "Diagnosis": {"TNBC": "BC"},   # {child: parent} for this field
        # fields absent here get no partial matching (default behavior)
    },
)

When a prediction equals the parent of the labeled child, the case scores as a Partial (Par) worth 0.5 in precision and recall rather than Incorrect:

A single case that is purely a parent-hit therefore scores 0.5 precision, 0.5 recall → 0.5 F1.

Two semantic properties:

• One level only — only a direct parent counts; a grandparent scores as Incorrect.
• Direction matters — credit is asymmetric: it is granted only when the prediction is the parent of the label (less specific than the truth), not when the prediction is more specific (a child) than the label.

Output: for each field that has a hierarchy entry, results get a per-row Par: {Field} column and the metrics table gets a par column (aggregated partial count). Fields without a hierarchy entry produce identical output to a run with no hierarchy at all (no Par:/Inc: columns).

Bootstrap Confidence Intervals

The framework includes statistical confidence interval estimation using non-parametric bootstrap resampling at the case level. This provides uncertainty quantification for all validation metrics.

Usage

from llmvalidate import bootstrap_CI

# After running validation to get results_df
ci_results = bootstrap_CI(
    res_df=results_df,           # Results from validate() function
    fields=["diagnosis", "treatment"],  # Fields to analyze (or None for auto-detect)
    n_bootstrap=5000,            # Number of bootstrap samples (default: 5000)
    ci=0.95,                     # Confidence level (default: 0.95 for 95% CI)
    random_state=42              # For reproducible results
)

Bootstrap Method

• Resampling unit: Individual cases (not individual predictions)
• Resampling strategy: Sample with replacement to preserve original dataset size
• CI calculation: Percentile method using bootstrap distribution
• Partial labeling: Handles missing labels gracefully - cases with missing labels for specific fields are excluded from calculations for those fields only
• Metrics included: All validation metrics (precision, recall, F1, accuracy, etc.)

Output Format

The bootstrap_CI() function returns a DataFrame with confidence intervals for each field:

Example output:

        field  labeled cases  precision (micro): mean  precision (micro): lower  precision (micro): upper
0  exceptions          1000                       NaN                       NaN                       NaN
1   diagnosis          1000                      0.82                      0.79                      0.85
2   treatment          1000                      0.91                      0.88                      0.94
3  N=5000; CI=95%       NaN                       NaN                       NaN                       NaN

The final row contains bootstrap parameters for reference: sample size (N) and confidence interval level (CI).

Use Cases

• Performance assessment: Quantify uncertainty in reported metrics
• Model comparison: Determine if performance differences are statistically significant
• Sample size planning: Understand precision of estimates with current dataset size
• Publication: Report confidence intervals alongside point estimates

🛠️ Advanced Configuration

Derived Fields (scored, but not an input column)

Every name in fields must normally be a column in source_df — that check is what catches a misspelled field name. But fields conflates two things: the column holding the labels and the name of the thing being scored. They usually coincide, and sometimes they don't: a comparison_callback can derive what it scores from a different column and write the field plus its own counts onto the row. Such a field has no input column to require.

Declare those names in derived_fields to exempt them from the check (and only them):

def comparison_callback(row, i, raw_text_column_name):
    # gold labels live in 'labels'; the scored 'spans' field is derived from them here
    golden_spans = spans_from(row["labels"])
    row["spans"] = golden_spans
    row["TP: spans"] = ...
    row["FP: spans"] = ...
    row["FN: spans"] = ...

results_df, metrics_df = validate(
    source_df=df,                      # has a 'labels' column, but no 'spans' column
    fields=["spans"],
    structure_callback=my_callback,
    comparison_callback=comparison_callback,
    derived_fields=("spans",),         # 'spans' is created during scoring, not read from input
)

Default () means no derived fields — every scored field must be an input column, as before. Each rule fails fast with a clear message:

Parallel Processing

validate(
    source_df=df,
    fields=["diagnosis", "treatment"], 
    structure_callback=callback,
    max_workers=None,      # Auto-detect CPU count (or specify number)
    use_threads=True       # True for I/O-bound (LLM API calls), False for CPU-bound
)

Performance Features

• Automatic caching - Identical raw text inputs are deduplicated and cached
• Progress tracking - Real-time progress bar for long-running validations
• Cache statistics - Check Sys: from cache column in results to monitor cache hits

Confidence Analysis

When LLM inference returns both extracted fields and their associated confidence levels, the framework automatically detects Res: {Field} confidence columns and generates:

• Separate metrics for each unique confidence level found in your data
• Overall metrics aggregating across all confidence levels
• Useful for setting confidence thresholds and analyzing prediction reliability

🧪 Development & Testing

# Install development dependencies
pip install -r requirements.txt

# Run all tests
pytest  

# Run with coverage reporting
pytest --cov=llmvalidate

# Run specific test modules
pytest tests/validate_test.py              # Core validation logic
pytest tests/compare_results_test.py       # Comparison algorithms  
pytest tests/compare_results_all_test.py   # End-to-end comparisons

📁 Project Structure

llm-validation-framework/
├── src/
│   └── llmvalidate/
│       ├── validation.py     # Main validation pipeline and metrics calculation
│       ├── structured.py     # Pydantic data models for LLM results
│       └── utils.py         # Utility functions (list conversion, flattening)
├── tests/               # Comprehensive test suite
├── validation_results/  # Output directory (auto-created)
├── samples.csv         # Demo dataset with all validation scenarios  
├── runme.py           # Demo script
└── requirements.txt   # Dependencies (pandas, pydantic, tqdm, etc.)

🔧 Troubleshooting

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.