schemafit

Provider-aware structured-output / JSON-Schema CI linter. Catch the schema incompatibilities that make one provider 400 while another succeeds, before they hit production, as a fast, offline CI check.

A JSON Schema / tool definition / response_format that works on OpenAI can 400 on Anthropic or Gemini (and vice-versa): nested oneOf, a missing additionalProperties: false, a default in a property, Anthropic-rejected validation keywords (minLength, format, pattern, …), Gemini's lack of anyOf/dict support. The API tells you it failed but not which constraint violated it, so teams hand-port schemas and debug by trial-and-error at runtime.

schemafit encodes each provider's documented constraint surface as a versioned, declarative rule pack and lints your schema statically, pointing at the exact JSON-Pointer path, the keyword, and why, with a non-zero exit code so CI fails the PR instead of prod.

Every rule is grounded in a real, cited provider issue (see schemafit/rules/). It is not a runtime client: it makes no model calls, needs no API key, and has zero runtime dependencies.

Try it in 10 seconds (no setup, no API key)

Paste this into a terminal. It needs uv, whose uvx runs schemafit in a throwaway environment. The schema below is valid for OpenAI strict mode, but it uses two keywords that other providers reject:

echo '{"type":"object","additionalProperties":false,"properties":{"sku":{"type":"string","pattern":"^[A-Z]{3}-[0-9]{4}$"},"qty":{"type":"integer","minimum":1}},"required":["sku","qty"]}' \
  | uvx schemafit lint - --provider openai,anthropic,gemini,mistral,cohere

Output from schemafit 0.5.1 (exit code 1, so a CI job fails here). Version 0.5.0 prints the same findings with a different separator after PASS/FAIL:

[openai] PASS: 0 error(s), 0 warning(s)
[anthropic] FAIL: 2 error(s), 0 warning(s)
  ERROR   #/properties/qty/minimum  (anthropic-no-minimum)
          Anthropic rejects the 'minimum' validation keyword (400 Bad Request).
          ref: https://github.com/vercel/ai/issues/13355
  ERROR   #/properties/sku/pattern  (anthropic-no-pattern)
          Anthropic rejects the 'pattern' validation keyword (400 Bad Request).
          ref: https://github.com/vercel/ai/issues/13355
[gemini] PASS: 0 error(s), 0 warning(s)
[mistral] PASS: 0 error(s), 0 warning(s)
[cohere] FAIL: 1 error(s), 0 warning(s)
  ERROR   #/properties/qty/minimum  (cohere-no-minimum)
          Cohere structured outputs do not support the 'minimum' numeric range keyword.
          ref: https://docs.cohere.com/docs/structured-outputs

No uv? pip install schemafit (Python 3.11+), then run the same line without uvx. To get a provider-valid variant, swap lint ... --provider ... for repair - --provider anthropic; it prints the fixed schema and counts which fixes drop a constraint (lossy=).

Proof: 88% of real LLM schemas would be rejected somewhere

I ran schemafit across 50 real, public structured-output / tool schemas from the OpenAI & Anthropic cookbooks, popular agent frameworks (instructor, langchain, pydantic-ai, crewai, …), and official MCP servers. 44 of 50 (88%) would be rejected by at least one major provider's constraints, and only 3 passed all five. The #1 culprit: additionalProperties, the field almost nobody sets, which OpenAI strict Structured Outputs requires to be false. That rule drives the headline: the 44 are exactly the schemas OpenAI strict mode rejects. If you don't use strict mode, the numbers that matter are Anthropic (6 of 50) and Cohere (13 of 50).

Full breakdown, per-provider numbers, and how to reproduce it on your own schemas: benchmarks/.

Why this and not Instructor / BAML / LiteLLM / Vercel AI SDK?

Those are excellent runtime clients: they normalize, repair, or constrain a schema at call-time. schemafit fills the gap they leave: a static, pre-ship CI lint that fails the build before the schema ever reaches a provider, over the raw schemas you already ship, with no DSL or codegen buy-in.

Install

pip install schemafit     # Python 3.11+, zero runtime dependencies
schemafit demo            # offline self-check; ends with PROOF_OK

Other ways to run it:

uvx schemafit --help                                             # no install (needs uv)
docker run --rm ghcr.io/orionarchitekton/schemafit demo          # container image
pip install "git+https://github.com/OrionArchitekton/schemafit"  # unreleased main branch

Usage

# Lint one schema against several providers (exit 1 if any error):
schemafit lint my-schema.json --provider openai,anthropic,gemini,mistral,cohere

# Machine-readable output for CI annotations:
schemafit lint my-schema.json --provider anthropic --format json

# SARIF 2.1.0 for GitHub code-scanning / the Security tab:
schemafit lint my-schema.json --provider openai,anthropic --format sarif > schemafit.sarif

# Confirm against the live provider (opt-in; MOCK unless a key is in the env):
schemafit lint my-schema.json --provider openai --live-verify

# Also fail on warnings (e.g. Gemini $ref recursion risk):
schemafit lint my-schema.json --provider gemini --strict

# Emit a best-effort provider-valid variant (lossy transforms are flagged):
schemafit repair my-schema.json --provider anthropic --out fixed.json

# List supported providers / run a hermetic end-to-end proof:
schemafit providers
schemafit demo

Use in CI

GitHub Actions (this repo ships a composite action):

- uses: OrionArchitekton/schemafit@v0.5.0
  with:
    schema: schemas/tool.json
    providers: openai,anthropic,gemini

Or directly / as a pre-commit hook (.pre-commit-hooks.yaml is included):

- repo: https://github.com/OrionArchitekton/schemafit
  rev: v0.5.0
  hooks:
    - id: schemafit
      args: ["--provider", "openai,anthropic,gemini"]
      files: '^schemas/.*\.json$'   # scope to YOUR LLM schemas, not every .json

Scope the hook with files: to the directory holding your LLM schemas; the default types: [json] would otherwise lint every JSON file in the repo (package.json, tsconfig.json, lockfiles), which are not LLM schemas.

GitHub code-scanning (SARIF)

--format sarif emits SARIF 2.1.0 so lint findings show up as annotations in the Security → Code scanning tab, with the exact JSON-Pointer path, the rule id, and the primary-source helpUri. SARIF is written to stdout regardless of the exit code, so code-scanning still ingests the artifact even when the gate fails (a clean schema produces a valid run with an empty results array, which clears stale alerts):

- run: schemafit lint schemas/*.json --provider openai,anthropic,gemini --format sarif > schemafit.sarif
  continue-on-error: true            # let code-scanning ingest the report
- uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: schemafit.sarif

Live verification (--live-verify, opt-in)

--live-verify turns "the docs forbid this" into "the provider actually accepted/rejected it" by sending a minimal structured-output probe to each provider and failing closed on a rejection. It is opt-in and key-gated:

• Default = MOCK: with no provider key in the environment it uses a deterministic, network-free client modeled on the static rule pack, so CI and the Docker image run it with no key and no network.
• Real call: only when the provider's key (OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY) is present in the environment. Real calls use the standard library (no new dependency); the [live] extra is reserved for optional provider SDKs.
• Tri-state: confirmed_by_provider is true (accepted), false (rejected → exit 1), or null (abstained: no key / rate-limited / network error). Abstaining is not a rejection and never fails CI.

Never commit an API key. The real path reads keys only from the environment and never echoes their value. Leave --live-verify out of default CI.

Supported providers

Exit codes

Scope and roadmap

In scope now: the lint + repair core, five provider rule packs (OpenAI, Anthropic, Gemini, Mistral, Cohere), human/JSON and SARIF 2.1.0 reporters, an opt-in --live-verify confirmation mode, Docker image, GitHub Action, pre-commit hook.

Shipped in v0.2: SARIF output for GitHub code-scanning; the --live-verify opt-in live-confirmation mode (MOCK by default, key-gated real calls, fail-closed).

Shipped in v0.3: the Mistral and Cohere provider rule packs (provider matrix 3 → 5), both static, network-free, and core-dependency-free.

Shipped in v0.4: Cohere structural rules (top-level must be object + every object ≥1 required field) via two new apply_rule kinds; primary-sourced, static, no net, core-deps unchanged.

Shipped in v0.5: automatic rule-pack drift detection on the --live-verify path (mock-default/hermetic): static pass + live reject emits *-drift finding (reuses Finding/report/SARIF + doc_url foundation). Proofs include schemafit lint fixtures/drift-mock-bad.json --provider cohere --live-verify (exit 1 + cohere-drift).

Deferred (vNext): Bedrock/Vertex packs; a pydantic source-model auto-fix mode; and an npm/ajv + Zod port.

License

MIT © 2026 Dan Mercede