valjson

Diagnostics for structured JSON output from LLMs: extract, validate, fix, score per-field accuracy, and gate on confidence. Most commands need no model.

Fine-tuning your LLM for JSON? Your aggregate metrics might be hiding per-field regressions.

pip install valjson

What it does

valjson is the observability layer for structured JSON output. It meets you wherever you are:

The problem

Standard fine-tuning + grammar-constrained decoding produces valid JSON. Aggregate loss improves. But:

STRUCTURAL         5.33 -> 0.00     -100%   OK
KEY                0.47 -> 0.00     -100%   OK
BOOLEAN            0.46 -> 1.05     +130%   !! REGRESSION
TOTAL              0.55 -> 0.17      -69%

Aggregate loss improved 69%. Boolean prediction got 130% worse. valjson catches this.

Per-field accuracy vs gold

When you have human-labeled gold JSONs, --compare gives you per-field accuracy without needing a model — role-aware, so a blown free-text field does not drown the signal on the constrained fields you actually care about:

valjson --compare \
    --schema schema.json \
    --data generated.jsonl \
    --gold gold.jsonl \
    --ignore-role STRING,ARRAY

• --match-by <key> — pair records by ID (wrapper-first lookup), not line order.
• --ignore-role STRING,ARRAY — focus on BOOLEAN / ENUM / NUMBER fields where exact equality is meaningful. Unmatched IDs are reported separately.

Evidential gating: abstain when the model is unsure

When the model returns a probability distribution over allowed values for each constrained field, --gate decides per record whether to commit, abstain, or reject based on the margin between the top two values:

valjson --gate \
    --data probs.jsonl \
    --margin-threshold 0.30

Input format (one record per line):

{"id": "rec-001",
 "probs": {
   "refundable": {"True": 0.553, "False": 0.447},
   "status":     {"submitted": 0.9, "pending": 0.1, "draft": 0.0}
 }}

A field's margin = top_prob − second_prob. If margin ≥ threshold, the field is committed; otherwise it abstains. Per-field abstention rates are reported — fields with persistently high abstention are diagnostic of underspecified training targets (the regression pattern this paper documents).

Quick start

See QUICK_START.md for a hands-on walkthrough from messy output to full analysis.

Python API

from valjson import analyze

report = analyze(
    model_name="Qwen/Qwen2.5-0.5B-Instruct",
    checkpoint="my_lora/",
    schema="schema.json",
    data="test.jsonl",
)
print(report)

if report.regressions:
    print(f"REGRESSIONS: {[r.role for r in report.regressions]}")

Exit code is 1 if regressions are detected. Use in CI/CD.

Grammar Roles

Links

• Quick Start Tutorial
• Fixing JSON Fine-Tuning — mitigations, services
• Paper — "Valid JSON, Wrong Answer" (2026), DOI 10.5281/zenodo.20075999

License

MIT