ai-qa-sdk 0.2.0

Universal AI validation SDK — hallucination, factuality, safety checks for any LLM output

ai-qa-sdk

Python SDK for validating AI outputs.

Installation

pip install ai-qa-sdk

This installs everything you need including support for Anthropic (Claude) and OpenAI (GPT).

Quick Start

1. Deterministic Checks (No API Key Needed)

from ai_qa import validate

# Validate JSON format - works immediately!
result = validate(
    output='{"name": "John", "age": 30}',
    checks=["json_format"]
)

print(f"Passed: {result.passed}")
for check in result.checks:
    print(f"  {check.name}: {check.score:.2f}")

2. LLM-Powered Checks (Configure Your Provider)

Get an API key first:

• Anthropic (Claude): https://console.anthropic.com → API Keys
• OpenAI (GPT): https://platform.openai.com/api-keys

Then configure and use:

from ai_qa import configure, validate

# Configure your preferred provider
configure(provider="anthropic", api_key="sk-ant-YOUR-KEY-HERE")

# Or use OpenAI instead
# configure(provider="openai", api_key="sk-YOUR-KEY-HERE")

# Now you can use any LLM-powered check
result = validate(
    output="The capital of France is Paris",
    input="What is the capital of France?",
    checks=["factuality", "relevance", "hallucination"]
)

print(f"Passed: {result.passed}")
print(f"Cost: ${result.total_cost:.4f}")

for check in result.checks:
    print(f"  {check.name}: {check.score:.2f}")

Available Checks

✓ No API Key Required

• json_format - Validate JSON syntax
• regex - Pattern matching
• schema - JSON schema validation
• length - Output length bounds

🔑 Requires API Key (Anthropic or OpenAI)

• hallucination - Detect made-up facts
• factuality - Check factual accuracy
• relevance - Relevance to input
• toxicity - Harmful content detection
• bias - Bias/fairness issues
• pii - Personal information detection
• consistency - Internal consistency
• contradiction - Logical contradictions
• groundedness - Grounded in provided context
• And more!

Providers

ai-qa-sdk supports multiple providers out of the box:

Provider	Setup	Cost
Anthropic (Claude)	configure(provider="anthropic", api_key="...")	~$0.003-0.015 per 1K tokens
OpenAI (GPT)	configure(provider="openai", api_key="...")	~$0.03-0.06 per 1K tokens
Local (Ollama)	configure(provider="local", base_url="http://localhost:11434")	Free (runs locally)

Get API keys:

• Anthropic: https://console.anthropic.com/api-keys
• OpenAI: https://platform.openai.com/api-keys