llm-comply
Multi-format LLM API compliance testing tool — validates endpoints against official specs for OpenAI Chat Completions, Open Responses, Anthropic Messages, and Google GenAI.
Install
pip install llm-comply
# For colored terminal output:
pip install llm-comply[rich]
Quick Start
# Open Responses (default)
llm-comply -u https://api.openai.com/v1 -k $OPENAI_API_KEY -m gpt-4o-mini
# OpenAI Chat Completions
llm-comply --format openai-chat -u https://api.openai.com/v1 -k $OPENAI_API_KEY -m gpt-4o-mini
# Anthropic Messages
llm-comply --format anthropic -u https://api.anthropic.com/v1 -k $KEY -m claude-haiku-4-5 \
--auth-header x-api-key --no-bearer -H anthropic-version:2023-06-01
# Google GenAI
llm-comply --format google-genai -u https://generativelanguage.googleapis.com -k $KEY \
-m gemini-2.5-flash --auth-header x-goog-api-key --no-bearer
Options
-u, --base-url URL API base URL (required)
-k, --api-key KEY API key (or set OPENRESPONSES_API_KEY env)
-m, --model MODEL Model name (default: gpt-4o-mini)
--format FORMAT API format: open-responses, openai-chat, anthropic, google-genai
-f, --filter IDS Comma-separated test IDs to run
-i, --ignore PATTERNS Ignore errors matching substrings (e.g. refusal,verbosity)
-H, --header K:V Extra headers (e.g. anthropic-version:2023-06-01)
--auth-header NAME Auth header name (default: Authorization)
--no-bearer Don't prepend "Bearer " to API key
-v, --verbose Show request/response on failure
--json JSON output for CI
--list List available test IDs
Test Coverage
| Format | Tests | What's Validated |
|---|---|---|
| Open Responses | 12 | Schema + semantic (lifecycle, phases, compaction) |
| OpenAI Chat | 8 | Schema + semantic (choices, finish_reason, delta) |
| Anthropic Messages | 8 | Schema + semantic (content blocks, stop_reason) |
| Google GenAI | 8 | Semantic only (candidates, parts, finishReason) |
License
MIT
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
llm_comply-0.3.0.tar.gz
(378.2 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
llm_comply-0.3.0-py3-none-any.whl
(387.5 kB
view details)
File details
Details for the file llm_comply-0.3.0.tar.gz.
File metadata
- Download URL: llm_comply-0.3.0.tar.gz
- Upload date:
- Size: 378.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
df2dc59d7f1579800e127ce9f78318c11427ae7a00869be114151d23fe262a20
|
|
| MD5 |
e31464d47ed20911941aa4fe2209374b
|
|
| BLAKE2b-256 |
499fbf5f7489db9067669db4985cba53bb6ddd50563b989cdd002b2d4bcc1b97
|
Provenance
The following attestation bundles were made for llm_comply-0.3.0.tar.gz:
Publisher:
release.yml on Oaklight/llm-comply
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
llm_comply-0.3.0.tar.gz -
Subject digest:
df2dc59d7f1579800e127ce9f78318c11427ae7a00869be114151d23fe262a20 - Sigstore transparency entry: 2387455990
- Sigstore integration time:
-
Permalink:
Oaklight/llm-comply@28cac2d456eaa8fadd774a3d30763524c7405b37 -
Branch / Tag:
refs/heads/master - Owner: https://github.com/Oaklight
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@28cac2d456eaa8fadd774a3d30763524c7405b37 -
Trigger Event:
workflow_dispatch
-
Statement type:
File details
Details for the file llm_comply-0.3.0-py3-none-any.whl.
File metadata
- Download URL: llm_comply-0.3.0-py3-none-any.whl
- Upload date:
- Size: 387.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
364551b1cbfdaa14c68737c9abc2ec7d95760f7e48a8e95a131c49d01a6dd9f2
|
|
| MD5 |
61b830a1bacbd391ce113bcb28df4072
|
|
| BLAKE2b-256 |
6fb30ac332480e7e628a9b12c39979bf50be47af41d389882bf62741961abcd4
|
Provenance
The following attestation bundles were made for llm_comply-0.3.0-py3-none-any.whl:
Publisher:
release.yml on Oaklight/llm-comply
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
llm_comply-0.3.0-py3-none-any.whl -
Subject digest:
364551b1cbfdaa14c68737c9abc2ec7d95760f7e48a8e95a131c49d01a6dd9f2 - Sigstore transparency entry: 2387456000
- Sigstore integration time:
-
Permalink:
Oaklight/llm-comply@28cac2d456eaa8fadd774a3d30763524c7405b37 -
Branch / Tag:
refs/heads/master - Owner: https://github.com/Oaklight
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@28cac2d456eaa8fadd774a3d30763524c7405b37 -
Trigger Event:
workflow_dispatch
-
Statement type:
