ai-qa-sdk 0.1.3

Universal AI validation SDK — hallucination, factuality, safety checks for any LLM output

ai-qa-sdk

Python SDK for validating AI outputs.

Installation

pip install ai-qa-sdk

Quick Start

1. Without API Key (Deterministic Checks)

from ai_qa import validate

# Validate JSON format (no API key needed)
result = validate(
    output='{"name": "John", "age": 30}',
    checks=["json_format"]
)

print(f"Passed: {result.passed}")
for check in result.checks:
    print(f"  {check.name}: {check.score:.2f}")

2. With API Key (LLM-Powered Checks)

Install with provider support:

# For Anthropic (Claude)
pip install ai-qa-sdk[anthropic]

# For OpenAI (GPT)
pip install ai-qa-sdk[openai]

# For all providers
pip install ai-qa-sdk[all-providers]

Configure and validate:

from ai_qa import configure, validate

# Configure Anthropic (Claude)
configure(provider="anthropic", api_key="sk-ant-...")

# Or configure OpenAI
# configure(provider="openai", api_key="sk-...")

# Now use LLM-powered checks
result = validate(
    output="The capital of France is Paris",
    input="What is the capital of France?",
    checks=["factuality", "relevance", "hallucination"]
)

print(f"Passed: {result.passed}")
print(f"Cost: ${result.total_cost:.4f}")

for check in result.checks:
    print(f"  {check.name}: {check.score:.2f}")

Available Checks

✓ No API Key Required

• json_format - Validate JSON syntax
• regex - Pattern matching
• schema - JSON schema validation
• length - Output length bounds

🔑 Requires API Key (Anthropic or OpenAI)

• hallucination - Detect made-up facts
• factuality - Check factual accuracy
• relevance - Relevance to input
• toxicity - Harmful content detection
• bias - Bias/fairness issues
• pii - Personal information detection
• consistency - Internal consistency
• contradiction - Logical contradictions
• groundedness - Grounded in provided context
• And more!

Getting API Keys

Anthropic (Claude):

• Visit: https://console.anthropic.com
• Create account → API Keys → Create new key
• Format: sk-ant-...

OpenAI (GPT):

• Visit: https://platform.openai.com/api-keys
• Create account → API Keys → Create new key
• Format: sk-...

Documentation

See CLAUDE.md for development guide.