RAIT Connector

Python library for evaluating LLM outputs across multiple ethical dimensions and performance metrics using Azure AI Evaluation services.

Features

• 22 Evaluation Metrics across 8 ethical dimensions
• Parallel Execution for faster evaluations
• Automatic API Integration with RAIT services
• Type-Safe with Pydantic models
• Flexible Configuration via environment variables or direct parameters
• Batch Processing with custom callbacks
• Scheduler for recurring telemetry, calibration, and bias evaluation jobs
• Bias Evaluation — automatic fairness checks using real traffic queries and RAIT-managed demographic templates
• Comprehensive Documentation with examples

Installation

pip install rait-connector

Or with uv:

uv add rait-connector

Quick Start

from rait_connector import RAITClient

# Initialize client
client = RAITClient()

# Evaluate a single prompt
result = client.evaluate(
    prompt_id="123",
    prompt_url="https://example.com/123",
    timestamp="2025-12-11T10:00:00+00:00",
    model_name="gpt-4",
    model_version="1.0",
    query="What is AI?",
    response="AI is artificial intelligence...",
    environment="production",
    purpose="monitoring"
)

print(f"Evaluation complete: {result['prompt_id']}")

Configuration

Environment Variables

Set required environment variables:

# RAIT API
export RAIT_API_URL="https://api.raitracker.com"
export RAIT_CLIENT_ID="your-client-id"
export RAIT_CLIENT_SECRET="your-client-secret"

# Azure OpenAI
export AZURE_OPENAI_ENDPOINT="https://your.openai.azure.com"
export AZURE_OPENAI_API_KEY="your-api-key"
export AZURE_OPENAI_DEPLOYMENT="your-deployment"
export AZURE_OPENAI_API_VERSION="2024-12-01-preview"  # optional, this is the default

# Azure AD
export AZURE_CLIENT_ID="your-azure-client-id"
export AZURE_TENANT_ID="your-azure-tenant-id"
export AZURE_CLIENT_SECRET="your-azure-client-secret"

# Azure Resources
export AZURE_SUBSCRIPTION_ID="your-subscription-id"
export AZURE_RESOURCE_GROUP="your-resource-group"
export AZURE_PROJECT_NAME="your-project-name"
export AZURE_ACCOUNT_NAME="your-account-name"
export AZURE_AI_PROJECT_URL="https://your.ai.azure.com/..."  # optional
export AZURE_LOG_ANALYTICS_WORKSPACE_ID="your-workspace-id"  # optional, for telemetry queries

# RAIT Ingest
export RAIT_INGEST_URL="https://your-ingest-endpoint"  # required — all log types route through here

Direct Configuration

Or pass configuration directly:

client = RAITClient(
    rait_api_url="https://api.raitracker.com",
    rait_client_id="your-client-id",
    rait_client_secret="your-secret",
    azure_openai_endpoint="https://your.openai.azure.com",
    azure_openai_api_key="your-key",
    azure_openai_deployment="gpt-4",
    # ... other parameters
)

Evaluation Metrics

RAIT Connector supports 22 metrics across 8 ethical dimensions:

Batch Evaluation

Evaluate multiple prompts efficiently:

prompts = [
    {
        "prompt_id": "001",
        "prompt_url": "https://example.com/001",
        "timestamp": "2025-12-11T10:00:00+00:00",
        "model_name": "gpt-4",
        "model_version": "1.0",
        "query": "What is AI?",
        "response": "AI is...",
        "environment": "production",
        "purpose": "monitoring"
    },
    # ... more prompts
]

summary = client.evaluate_batch(prompts)
print(f"Completed: {summary['successful']}/{summary['total']}")

With Custom Callback

def on_complete(summary):
    print(f"Success: {summary['successful']}")
    print(f"Failed: {summary['failed']}")

client.evaluate_batch(prompts, on_complete=on_complete)

Calibration

Automatic Background Calibration

When you call evaluate(), the client automatically:

1. Checks the API for calibration prompts
2. If available, runs calibration in the background (once per model/version/environment)
3. Evaluates calibration prompts with pre-defined responses

This happens automatically - no manual intervention needed!

Scheduler

Run recurring telemetry and calibration jobs automatically:

from rait_connector import RAITClient, Scheduler

client = RAITClient()
scheduler = Scheduler(client)

scheduler.add_telemetry_job(
    model_name="gpt-4",
    model_version="1.0",
    model_environment="production",
    model_purpose="monitoring",
    interval="daily",
)
scheduler.add_calibration_job(
    model_name="gpt-4",
    model_version="1.0",
    environment="production",
    model_purpose="monitoring",
    invoke_model=lambda prompt: my_llm(prompt),
    interval="weekly",
)
scheduler.add_bias_job(
    model_name="gpt-4",
    model_version="1.0",
    environment="production",
    model_purpose="monitoring",
    invoke_model=lambda prompt: my_llm(prompt),
    interval="weekly",
)

scheduler.start()  # runs in background

# Inspect job state
print(scheduler.status())   # registered jobs and next run time
print(scheduler.history())  # past execution records

Supports named intervals ("hourly", "daily", "weekly"), cron expressions, timedelta, or raw seconds. Custom jobs can be registered via add_job() or the @scheduler.job() decorator.

Parallel Execution

Control parallelism for faster evaluations:

result = client.evaluate(
    ...,
    parallel=True,
    max_workers=10  # Use 10 parallel workers
)

Documentation

Full documentation is published at https://responsible-systems.github.io/rait_connector/.

Versioned docs are available for each release (e.g. /0.6.0/). The latest alias always points to the most recent release.

Requirements

• Python 3.12+
• Azure OpenAI access
• RAIT API credentials