llmcontract

Cross-provider schema contract testing for LLMs.

Define the expected shape of an LLM response once. Validate it against OpenAI, Anthropic, Mistral, LiteLLM, or any model that returns JSON — with a single, consistent API.

from llmcontract import Contract, field, required_fields

contract = Contract(
    required_fields("action", "confidence", "reasoning"),
    field("action",     type=str,   values=["buy", "sell", "hold"]),
    field("confidence", type=float),
    field("reasoning",  type=str,   min_len=20),
)

# Works with any provider
from llmcontract import openai_response
data = openai_response(client.chat.completions.create(...))
contract.validate(data)   # raises ContractViolation on mismatch

---

Why llmcontract?

Pain	llmcontract fix
Provider A returns {"action": "BUY"}, provider B returns {"signal": "buy"}	One contract definition catches both regressions
LLM adds unexpected keys that break downstream code	no_extra_fields() validator
Confidence field is sometimes a string, sometimes a float	field_type() catches it every time
Nested tool-call responses are hard to validate	nested() validator recurses cleanly
You switch providers and don't know what broke	Run the same contract against the new provider

---

Install

pip install llmcontract

No hard dependencies. Provider SDKs are optional:

pip install "llmcontract[openai]"
pip install "llmcontract[anthropic]"
pip install "llmcontract[all]"

---

Quickstart

1. Define a contract

from llmcontract import Contract, field, required_fields, no_extra_fields

trade_contract = Contract(
    required_fields("action", "confidence"),
    field("action",     type=str,   values=["buy", "sell", "hold"]),
    field("confidence", type=float),
    field("reasoning",  type=str,   min_len=10, required=False),
    no_extra_fields("action", "confidence", "reasoning"),
    name="TradeSignal",   # optional, shown in error messages
)

2. Validate an OpenAI response

import openai
from llmcontract import openai_response

client = openai.OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Give me a trade signal as JSON."}],
)

data = openai_response(response)
trade_contract.validate(data)

3. Validate an Anthropic response

import anthropic
from llmcontract import anthropic_response

client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-opus-4-6",
    max_tokens=256,
    messages=[{"role": "user", "content": "Give me a trade signal as JSON."}],
)

data = anthropic_response(response)
trade_contract.validate(data)

4. Validate a raw JSON string

from llmcontract import raw_json

text = '```json\n{"action": "buy", "confidence": 0.87}\n```'
data = raw_json(text)          # strips fences, parses JSON
trade_contract.validate(data)

---

Validators

Validator	What it checks
required_fields(*names)	All named fields exist
field_type(name, type)	Field is a specific Python type
enum_values(name, allowed)	Field value is in a list of allowed values
min_length(name, n)	String/list has at least n chars/items
max_length(name, n)	String/list has at most n chars/items
regex_match(name, pattern)	String field matches a regex
no_extra_fields(*allowed)	Response has no unexpected keys
nested(name, contract)	Sub-dict satisfies a child contract
field(name, ...)	All of the above in one call

field() — the Swiss Army knife

field(
    "email",
    required=True,          # default True
    type=str,
    pattern=r"^[^@]+@[^@]+\.[^@]+$",
    min_len=5,
    max_len=254,
)

---

Strict mode — collect all violations

By default, validate() raises on the first failure. In strict mode it collects every failure and raises a single combined error:

contract = Contract(
    required_fields("a", "b", "c"),
    strict=True,
)

try:
    contract.validate({})
except ContractViolation as exc:
    print(exc)
    # Contract [MyContract] violated with 3 error(s):
    #   • Required field missing: 'a'
    #   • Required field missing: 'b'
    #   • Required field missing: 'c'

---

Extend a contract

base = Contract(required_fields("id", "name"))
strict_version = base.extend(
    field("id",   type=int),
    field("name", type=str, min_len=1),
    no_extra_fields("id", "name"),
)

---

Nested responses

from llmcontract import Contract, nested, required_fields, field

address_contract = Contract(
    required_fields("street", "city", "country"),
    field("country", type=str, min_len=2, max_len=2),  # ISO 3166-1 alpha-2
)

person_contract = Contract(
    required_fields("name", "address"),
    field("name", type=str),
    nested("address", address_contract),
)

person_contract.validate({
    "name": "Alice",
    "address": {"street": "42 Main St", "city": "Springfield", "country": "US"},
})

---

Use with pytest

# tests/test_my_llm.py
import pytest
from llmcontract import Contract, required_fields, field

@pytest.fixture
def contract():
    return Contract(
        required_fields("answer", "confidence"),
        field("confidence", type=float),
    )

def test_response_shape(contract, mock_llm_response):
    data = raw_json(mock_llm_response)
    contract.validate(data)   # pytest shows ContractViolation as assertion error

---

Exceptions

All exceptions inherit from ContractViolation(AssertionError):

• FieldMissingError — required field absent
• FieldTypeError — wrong Python type
• EnumViolationError — value not in allowed list
• LengthViolationError — string/list too short or too long
• PatternViolationError — regex did not match
• ExtraFieldError — unexpected keys present

---

Provider adapters

Adapter	Usage
openai_response(resp)	openai>=1.0 ChatCompletion objects
anthropic_response(resp)	anthropic>=0.20 Message objects
litellm_response(resp)	LiteLLM ModelResponse (OpenAI-compatible)
raw_json(text)	Raw JSON string, strips code fences
raw_dict(data)	Pass-through for already-parsed dicts

---

License

MIT