chaincheck-llm

Validate LLM responses with composable, production-ready checks.

pip install chaincheck-llm

The Problem

LLMs return unpredictable output. Sometimes they leak PII. Sometimes they refuse to answer. Sometimes they hallucinate. You need a way to validate responses before they reach your users.

chaincheck-llm gives you a composable validation pipeline — run any combination of checks on any LLM response.

Quick Start

from chaincheck import validate
from chaincheck.checks import (
    check_not_empty,
    check_no_pii,
    check_no_toxicity,
    check_min_length,
)

result = validate(
    response=llm_output,
    checks=[
        check_not_empty,
        check_no_pii,
        check_no_toxicity,
        lambda r: check_min_length(r, min_chars=50),
    ]
)

print(result.passed)    # True / False
print(result.score)     # 0.0 - 1.0
print(result.summary())

Available Checks

Format Checks (free, no LLM needed)

Content Checks (free, rule-based)

LLM-as-Judge Checks (uses tokens)

Composing Checks

from chaincheck import validate
from chaincheck.checks import (
    check_not_empty, check_no_pii, check_no_toxicity,
    check_relevance, check_min_length,
)

result = validate(
    response=llm_output,
    checks=[
        # Fast rule-based checks first
        check_not_empty,
        check_no_pii,
        check_no_toxicity,
        lambda r: check_min_length(r, min_chars=100),
        # LLM judge last (costs tokens)
        lambda r: check_relevance(
            r,
            question="What is machine learning?",
            provider="openai",
            model="gpt-4o-mini",
            api_key="sk-...",
        ),
    ],
    fail_fast=True,  # stop after first failure
)

if not result.passed:
    for check in result.failed_checks:
        print(f"FAILED: {check.name} — {check.message}")

ValidationResult

result.passed           # True if all checks passed
result.score            # Average score (0.0 - 1.0)
result.checks           # List of CheckResult
result.passed_checks    # Checks that passed
result.failed_checks    # Checks that failed
result.summary()        # Dict with full breakdown

License

MIT