valjson 0.3.0

Per-grammar-role loss decomposition for fine-tuned structured JSON output

valjson

Per-grammar-role loss analysis for structured JSON output from language models.

Fine-tuning your LLM for JSON? Your aggregate metrics might be hiding per-field regressions.

pip install valjson

What it does

valjson is the observability layer for structured JSON output. It meets you wherever you are:

You have	Command	Needs model?
Messy text with JSON	valjson --extract --data output.txt	No
JSON + schema	valjson --validate --schema s.json --data output.jsonl	No
Broken JSON + schema	valjson --fix --schema s.json --data output.jsonl	No
JSON + schema	valjson --anatomy --schema s.json --data output.jsonl	No
Two output sets	valjson --diff --schema s.json --data a.jsonl --data2 b.jsonl	No
Model + checkpoint	valjson --checkpoint lora/ --schema s.json --data test.jsonl	Yes

The problem

Standard fine-tuning + grammar-constrained decoding produces valid JSON. Aggregate loss improves. But:

STRUCTURAL         5.33 -> 0.00     -100%   OK
KEY                0.47 -> 0.00     -100%   OK
BOOLEAN            0.46 -> 1.05     +130%   !! REGRESSION
TOTAL              0.55 -> 0.17      -69%

Aggregate loss improved 69%. Boolean prediction got 130% worse. valjson catches this.

Quick start

See QUICK_START.md for a hands-on walkthrough from messy output to full analysis.

Python API

from valjson import analyze

report = analyze(
    model_name="Qwen/Qwen2.5-0.5B-Instruct",
    checkpoint="my_lora/",
    schema="schema.json",
    data="test.jsonl",
)
print(report)

if report.regressions:
    print(f"REGRESSIONS: {[r.role for r in report.regressions]}")

Exit code is 1 if regressions are detected. Use in CI/CD.

Grammar Roles

Role	Description	Examples
STRUCTURAL	JSON syntax	{ } [ ] : ,
QUOTE	String delimiters	"
KEY	Object key characters	city, cuisine
ENUM_VALUE	Categorical values	Italian, Economy
BOOLEAN	Boolean strings	True, False
NUMBER	Numeric characters	42, 3.14
FREE_TEXT	Non-categorical content	names, addresses
WHITESPACE	Formatting	spaces, newlines

Links

• Quick Start Tutorial
• Fixing JSON Fine-Tuning — mitigations, services
• Paper — "Valid JSON, Wrong Answer" (2026)

License

MIT