rait-connector 0.6.0

Python library for evaluating LLM outputs across multiple ethical dimensions and performance metrics using Azure AI Evaluation services.

RAIT Connector

Python library for evaluating LLM outputs across multiple ethical dimensions and performance metrics using Azure AI Evaluation services.

Features

• 22 Evaluation Metrics across 8 ethical dimensions
• Parallel Execution for faster evaluations
• Automatic API Integration with RAIT services
• Type-Safe with Pydantic models
• Flexible Configuration via environment variables or direct parameters
• Batch Processing with custom callbacks
• Scheduler for recurring telemetry and calibration jobs
• Comprehensive Documentation with examples

Installation

pip install rait-connector

Or with uv:

uv add rait-connector

Quick Start

from rait_connector import RAITClient

# Initialize client
client = RAITClient()

# Evaluate a single prompt
result = client.evaluate(
    prompt_id="123",
    prompt_url="https://example.com/123",
    timestamp="2025-12-11T10:00:00Z",
    model_name="gpt-4",
    model_version="1.0",
    query="What is AI?",
    response="AI is artificial intelligence...",
    environment="production",
    purpose="monitoring"
)

print(f"Evaluation complete: {result['prompt_id']}")

Configuration

Environment Variables

Set required environment variables:

# RAIT API
export RAIT_API_URL="https://api.raitracker.com"
export RAIT_CLIENT_ID="your-client-id"
export RAIT_CLIENT_SECRET="your-client-secret"

# Azure OpenAI
export AZURE_OPENAI_ENDPOINT="https://your.openai.azure.com"
export AZURE_OPENAI_API_KEY="your-api-key"
export AZURE_OPENAI_DEPLOYMENT="your-deployment"
export AZURE_OPENAI_API_VERSION="2024-12-01-preview"  # optional, this is the default

# Azure AD
export AZURE_CLIENT_ID="your-azure-client-id"
export AZURE_TENANT_ID="your-azure-tenant-id"
export AZURE_CLIENT_SECRET="your-azure-client-secret"

# Azure Resources
export AZURE_SUBSCRIPTION_ID="your-subscription-id"
export AZURE_RESOURCE_GROUP="your-resource-group"
export AZURE_PROJECT_NAME="your-project-name"
export AZURE_ACCOUNT_NAME="your-account-name"
export AZURE_AI_PROJECT_URL="https://your.ai.azure.com/..."  # optional
export AZURE_LOG_ANALYTICS_WORKSPACE_ID="your-workspace-id"  # optional, for telemetry queries

# RAIT Ingest
export RAIT_INGEST_URL="https://your-ingest-endpoint"  # required — all log types route through here

Direct Configuration

Or pass configuration directly:

client = RAITClient(
    rait_api_url="https://api.raitracker.com",
    rait_client_id="your-client-id",
    rait_client_secret="your-secret",
    azure_openai_endpoint="https://your.openai.azure.com",
    azure_openai_api_key="your-key",
    azure_openai_deployment="gpt-4",
    # ... other parameters
)

Evaluation Metrics

RAIT Connector supports 22 metrics across 8 ethical dimensions:

Dimension	Metrics
Bias and Fairness	Hate and Unfairness
Explainability and Transparency	Ungrounded Attributes, Groundedness, Groundedness Pro
Monitoring and Compliance	Content Safety
Legal and Regulatory Compliance	Protected Materials
Security and Adversarial Robustness	Code Vulnerability
Model Performance	Coherence, Fluency, QA, Similarity, F1 Score, BLEU, GLEU, ROUGE, METEOR, Retrieval
Human-AI Interaction	Relevance, Response Completeness
Social and Demographic Impact	Sexual, Violence, Self-Harm

Batch Evaluation

Evaluate multiple prompts efficiently:

prompts = [
    {
        "prompt_id": "001",
        "prompt_url": "https://example.com/001",
        "timestamp": "2025-12-11T10:00:00Z",
        "model_name": "gpt-4",
        "model_version": "1.0",
        "query": "What is AI?",
        "response": "AI is...",
        "environment": "production",
        "purpose": "monitoring"
    },
    # ... more prompts
]

summary = client.evaluate_batch(prompts)
print(f"Completed: {summary['successful']}/{summary['total']}")

With Custom Callback

def on_complete(summary):
    print(f"Success: {summary['successful']}")
    print(f"Failed: {summary['failed']}")

client.evaluate_batch(prompts, on_complete=on_complete)

Calibration

Automatic Background Calibration

When you call evaluate(), the client automatically:

1. Checks the API for calibration prompts
2. If available, runs calibration in the background (once per model/version/environment)
3. Evaluates calibration prompts with pre-defined responses

This happens automatically - no manual intervention needed!

Scheduler

Run recurring telemetry and calibration jobs automatically:

from rait_connector import RAITClient, Scheduler

client = RAITClient()
scheduler = Scheduler(client)

scheduler.add_telemetry_job(
    model_name="gpt-4",
    model_version="1.0",
    model_environment="production",
    model_purpose="monitoring",
    interval="daily",
)
scheduler.add_calibration_job(
    model_name="gpt-4",
    model_version="1.0",
    environment="production",
    model_purpose="monitoring",
    invoke_model=lambda prompt: my_llm(prompt),
    interval="weekly",
)

scheduler.start()  # runs in background

# Inspect job state
print(scheduler.status())   # registered jobs and next run time
print(scheduler.history())  # past execution records

Supports named intervals ("hourly", "daily", "weekly"), cron expressions, timedelta, or raw seconds. Custom jobs can be registered via add_job() or the @scheduler.job() decorator.

Parallel Execution

Control parallelism for faster evaluations:

result = client.evaluate(
    ...,
    parallel=True,
    max_workers=10  # Use 10 parallel workers
)

Documentation

Full documentation is available in the docs/ directory:

• Installation Guide
• Quick Start
• API Reference
• Scheduler
• Examples

Requirements

• Python 3.12+
• Azure OpenAI access
• RAIT API credentials

Development

Setup

Clone the repository:

git clone https://github.com/Responsible-Systems/rait-connector.git
cd rait-connector

Install dependencies:

uv sync --dev

Install pre-commit hooks:

uv tool install pre-commit
pre-commit install

Project Documentation

Serve docs locally:

uv run mkdocs serve

Build docs:

uv run mkdocs build

Releasing a New Version

See CONTRIBUTING.md for the full release process.

In short: update CHANGELOG.md, bump the version, commit, tag with a v prefix, and push — CI handles the build, PyPI publish, and GitHub Release automatically.

Contributing

Contributions are welcome! See CONTRIBUTING.md for setup, branch naming, commit conventions, and the release process.

Support

For issues and questions:

• GitHub Issues: https://github.com/Responsible-Systems/rait-connector/issues

Changelog

See CHANGELOG.md for release history.