schemasplint 0.1.0

A Python library that validates and auto-repairs JSON output from LLMs against Pydantic schemas with pluggable business rules.

SchemaSplint

A lightweight, zero-dependency Python library that validates, auto-repairs, and enforces business rules on JSON outputs generated by Large Language Models (LLMs) such as Google Gemini, OpenAI GPT, and Anthropic Claude.

---

What It Does

schemasplint acts as a guardrail layer between raw LLM outputs and your application code. It ensures that JSON returned by AI models strictly conforms to expected structure and domain logic before your application consumes it.

• Schema Validation: Guarantees raw LLM JSON matches your expected Pydantic models.
• Categorized Error Reporting: Classifies failures cleanly into MissingField, TypeMismatch, MalformedJSON, and BusinessRuleViolation.
• Deterministic Auto-Repair: Fixes common LLM formatting mistakes (markdown fences, trailing commas, non-JSON conversational text, numeric string coercion) instantly without calling or paying for extra AI API calls.
• Pluggable Business Rules: Custom guardrail checks for domain constraints (e.g., preventing overlapping schedule slots, enforcing numeric range bounds).
• Embedded SQLite Logging: Records validation metrics (pass rate, latency in ms, error breakdown, auto-repairs) to a local SQLite database for observability.

---

Why SchemaSplint? (The Problem It Solves)

LLMs are probabilistic and frequently produce output flaws that break production applications:

1. Markdown Fences & Noise: Wrapping JSON in ```json ... ``` or prepending conversational greetings ("Here is your JSON:").
2. Invalid Syntax: Trailing commas before closing brackets or quotes in numeric fields.
3. Type Mismatches: Outputting strings like "42" when your code expects an integer 42.
4. Logical Violations: Producing syntactically valid JSON that violates domain rules (e.g., booking two overlapping appointments at 10:00 AM).

Instead of retrying expensive LLM calls or writing custom regex patches for every prompt, schemasplint deterministically repairs formatting errors and validates business constraints in less than a millisecond.

---

Installation

pip install schemasplint

(Requires Python 3.10+ and Pydantic v2+)

---

Quickstart

from pydantic import BaseModel
from schemasplint import Guard, Logger, OverlapRule, RangeRule

# 1. Define your Pydantic schema
class Task(BaseModel):
    label: str
    start_time: str
    end_time: str
    priority: int

class DailySchedule(BaseModel):
    user_id: int
    tasks: list[Task]

# 2. Configure guard with pluggable business rules & logging
logger = Logger("schemasplint.db")
guard = Guard(
    schema=DailySchedule,
    rules=[
        OverlapRule(items_field="tasks", start_field="start_time", end_field="end_time"),
        RangeRule(field="user_id", min_value=1),
    ],
    logger=logger,
)

# 3. Messy LLM JSON response string
raw_llm_response = """
Here is the requested schedule:
```json
{
    "user_id": "101",
    "tasks": [
        {"label": "Team Standup", "start_time": "09:00", "end_time": "09:30", "priority": "1"},
        {"label": "Deep Work", "start_time": "10:00", "end_time": "12:00", "priority": "2"},
    ]
}

"""

4. Validate with auto-repair enabled

result = guard.validate(raw_llm_response, auto_repair=True)

if result.success: print("Validation Succeeded!") print(f"User ID: {result.parsed_data.user_id}") print(f"Tasks Count: {len(result.parsed_data.tasks)}") if result.repaired: print("Auto-repair log:", result.repair_changes) else: print("Validation Failed!") for err in result.errors: print(f"[{err.error_type.value}] {err.field}: {err.message}")

5. Check SQLite validation summary metrics

summary = logger.get_summary() print(f"Pass Rate: {summary['pass_rate']}% across {summary['total_runs']} runs")

---

## Benchmark Results

Tested against 120 programmatically generated examples across 10 known LLM-output failure categories (markdown fences, trailing commas, conversational wrapper text, numeric-string type mismatches, missing fields, broken syntax, and business-rule violations).

- **Pass rate without repair:** 20.0%
- **Pass rate with SchemaSplint auto-repair:** 60.0%
- **Improvement:** +40.0 percentage points

![Benchmark results by category](benchmark/results_chart.png)

| Category | Raw Pass % | Repaired Pass % |
|---|---|---|
| broken_syntax | 0.0% | 0.0% |
| clean_valid | 100.0% | 100.0% |
| combined_messy | 0.0% | 100.0% |
| conversational_wrapper | 0.0% | 100.0% |
| markdown_fence | 0.0% | 100.0% |
| missing_field | 0.0% | 0.0% |
| numeric_string | 100.0% | 100.0% |
| overlap_violation | 0.0% | 0.0% |
| range_violation | 0.0% | 0.0% |
| trailing_comma | 0.0% | 100.0% |

*Categories like `missing_field`, `broken_syntax`, `overlap_violation`, and `range_violation` correctly stay at 0% even after repair — these represent cases where data is truly missing or a real business rule is broken, which a deterministic repair layer should never silently paper over rather than surface as an error.*

Reproduce this yourself:
```bash
python benchmark/generate_corpus.py
python benchmark/run_eval.py

---

License

MIT License