trustmodel 0.3.0

Official Python SDK for TrustModel AI evaluation platform

Official Python SDK for the TrustModel AI evaluation platform

Website • Documentation • Dashboard

---

Evaluate AI models for safety, bias, and performance with a simple, intuitive interface.

Features

• 🚀 Simple Interface: Easy-to-use client for all TrustModel operations
• 🔒 Secure: API key authentication with built-in validation
• 🎯 Type Safe: Full type hints for excellent IDE support
• 🔄 Reliable: Automatic retries and comprehensive error handling
• 📊 Comprehensive: Support for all evaluation types and configurations
• 🌍 Framework Agnostic: Works with any Python framework or standalone scripts

Installation

pip install trustmodel

Prerequisites

Before using the SDK, you must complete the following setup in the TrustModel Dashboard:

1. Create an API Key (Required)

You need a TrustModel API key to authenticate all SDK requests:

1. Go to Keys & Webhooks in the dashboard
2. Click "Create API Key"
3. Copy your new API key (starts with tm-)
4. Store it securely - you won't be able to see it again

2. Configure Webhooks (Required)

To receive notifications when evaluations complete or fail, you must configure webhooks:

1. Go to Keys & Webhooks in the dashboard
2. Click "Create Webhook"
3. Enter your webhook endpoint URL
4. Select the events you want to receive
5. Save your webhook configuration

Important: Without configuring both an API key and webhooks in the webapp, you cannot run evaluations. The API will return an error if these are not set up.

Quick Start

import trustmodel

# Initialize the client
client = trustmodel.TrustModelClient(api_key="tm-your-api-key-here")

# List available models
models, api_sources = client.models.list()
print(f"Found {len(models)} models available")

# Create an evaluation
evaluation = client.evaluations.create(
    model_identifier="gpt-4",
    vendor_identifier="openai",
    categories=["safety", "bias", "performance"]
)

print(f"Evaluation created with ID: {evaluation.id}")
print(f"Status: {evaluation.status}")

# You'll receive a webhook notification when the evaluation completes
# Then retrieve the results:
completed_evaluation = client.evaluations.get(evaluation.id)
print(f"Overall score: {completed_evaluation.overall_score}")

# Check your credit balance
credits = client.credits.get_balance()
print(f"Credits remaining: {credits.credits_remaining}")

Authentication

Get your API key from the TrustModel Dashboard and use it to initialize the client:

import trustmodel

client = trustmodel.TrustModelClient(api_key="tm-your-api-key-here")

For production applications, store your API key securely using environment variables:

import os
import trustmodel

api_key = os.getenv("TRUSTMODEL_API_KEY")
client = trustmodel.TrustModelClient(api_key=api_key)

Evaluation Modes

TrustModel supports three ways to evaluate AI models:

Mode	Use Case	API Key Required
Platform Key	Quick evaluations using TrustModel's API keys	No (uses TrustModel's keys)
BYOK	Use your own vendor API key for any model	Yes (your vendor API key)
Custom Endpoint	Evaluate private/self-hosted models	Yes (your endpoint's API key)

Getting Available Vendors

Use client.config.get().vendors to discover available vendors:

config = client.config.get()

# Public vendors - for Platform Key and BYOK evaluations
public_vendors = config.vendors["public"]
for vendor in public_vendors:
    print(f"{vendor['identifier']}: {vendor['name']}")

# Custom vendors - for Custom Endpoint evaluations only
custom_vendors = config.vendors["custom"]
for vendor in custom_vendors:
    print(f"{vendor['identifier']}: {vendor['name']}")

Vendor Type	Use With	Description
public	Platform Key, BYOK	Vendors like OpenAI, Anthropic, Google AI for standard evaluations
custom	Custom Endpoint	Validators for self-hosted/private endpoints (OpenAI-compatible, Hugging Face, Azure AI, etc.)

Getting Available Models

Use client.models.list() to discover available models:

# Get all available models and API source info
models, api_sources = client.models.list()

# List all models with their details
for model in models:
    print(f"Model: {model.name}")
    print(f"  Identifier: {model.model_identifier}")
    print(f"  Vendor: {model.vendor_identifier}")
    print(f"  Platform Key Available: {model.available_via_trust_model_key}")
    print(f"  BYOK Available: {model.available_via_byok}")

# Filter models by vendor
openai_models = [m for m in models if m.vendor_identifier == "openai"]

# Filter models available via platform key (no vendor API key needed)
platform_key_models = [m for m in models if m.available_via_trust_model_key]

# Use a model in evaluation
model = models[0]
evaluation = client.evaluations.create(
    model_identifier=model.model_identifier,
    vendor_identifier=model.vendor_identifier,
    categories=["safety", "bias"]
)

Model Field	Type	Description
name	str	Human-readable model name
model_identifier	str	Identifier to use in API calls
vendor_identifier	str	Vendor identifier
available_via_trust_model_key	bool	Can evaluate without vendor API key
available_via_byok	bool	Previously used with your own API key

Platform Key (Default)

Use TrustModel's platform keys for quick evaluations. No vendor API key needed:

evaluation = client.evaluations.create(
    model_identifier="gpt-4",
    vendor_identifier="openai",
    categories=["safety", "bias"]
)

Note: Platform key availability varies by model. Check model.available_via_trust_model_key to see if a model supports this mode.

BYOK (Bring Your Own Key)

Use your own vendor API key to evaluate any model. All vendors support BYOK:

evaluation = client.evaluations.create(
    model_identifier="gpt-4",
    vendor_identifier="openai",
    api_key="sk-your-openai-key",  # Your OpenAI API key
    categories=["safety", "bias"]
)

How it works:

1. You provide your vendor API key (e.g., OpenAI, Anthropic, Google)
2. TrustModel validates the key before creating the evaluation
3. If validation fails, a ConnectionValidationError is raised with details
4. Your key is securely stored and used for the evaluation

Getting vendor API keys:

• OpenAI: platform.openai.com/api-keys
• Anthropic: console.anthropic.com/settings/keys
• Google AI: aistudio.google.com/apikey

Example with error handling:

from trustmodel import ConnectionValidationError, InsufficientCreditsError

try:
    evaluation = client.evaluations.create(
        model_identifier="gpt-4",
        vendor_identifier="openai",
        api_key="sk-your-openai-key",
        categories=["safety", "bias"]
    )
    print(f"Evaluation created: {evaluation.id}")
except ConnectionValidationError as e:
    # API key validation failed
    print(f"Invalid API key: {e.message}")
    if e.validation_details:
        print(f"Details: {e.validation_details}")
except InsufficientCreditsError as e:
    print(f"Need more credits: {e.credits_required} required")

Custom Endpoint

Evaluate your own OpenAI-compatible API endpoint (Ollama, vLLM, LiteLLM, Azure AI, etc.):

# Create evaluation for a custom endpoint
evaluation = client.evaluations.create_custom_endpoint(
    api_endpoint="https://api.yourcompany.com/v1",
    api_key="your-api-key",
    model_identifier="your-model-id",
    vendor_identifier="openai",  # Determines which validator to use
    model_name="My Custom Model",  # Optional display name
    categories=["safety", "bias"]
)

Available vendor identifiers for custom endpoints:

Get the list programmatically with client.config.get().vendors["custom"], or use one of these:

Identifier	Use For
openai	OpenAI-compatible APIs (Ollama, vLLM, LiteLLM, etc.) - default
huggingface	Hugging Face Inference Endpoints
azure_ai	Azure AI / Azure OpenAI Service
xai	Google Vertex AI
bedrock	AWS Bedrock

Examples:

# Ollama endpoint (uses default "openai" validator)
evaluation = client.evaluations.create_custom_endpoint(
    api_endpoint="http://localhost:11434/v1",
    api_key="ollama",  # Ollama doesn't require a real key
    model_identifier="llama3:8b"
)

# Azure AI endpoint
evaluation = client.evaluations.create_custom_endpoint(
    api_endpoint="https://your-resource.openai.azure.com",
    api_key="your-azure-key",
    model_identifier="gpt-4",
    vendor_identifier="azure_ai"
)

# Hugging Face endpoint
evaluation = client.evaluations.create_custom_endpoint(
    api_endpoint="https://api-inference.huggingface.co/models/your-model",
    api_key="hf_your_token",
    model_identifier="your-model",
    vendor_identifier="huggingface"
)

---

Core Concepts

Models

Discover available AI models:

# List all available models
models, api_sources = client.models.list()

for model in models:
    print(f"Model: {model.name}")
    print(f"Vendor: {model.vendor_identifier}")
    print(f"Platform key available: {model.available_via_trust_model_key}")
    print(f"Previously used BYOK: {model.available_via_byok}")
    print("---")

# Get specific model
model = client.models.get("openai", "gpt-4")
print(f"Found model: {model.name}")

Note: available_via_byok indicates you have previously used BYOK for this vendor. All vendors support BYOK - you can use your own API key with any model.

Evaluations

Create and manage AI model evaluations:

# Platform key (default) - uses TrustModel's keys
evaluation = client.evaluations.create(
    model_identifier="gpt-4",
    vendor_identifier="openai",
    categories=["safety", "bias"]
)

# BYOK - uses your own API key
evaluation = client.evaluations.create(
    model_identifier="gpt-4",
    vendor_identifier="openai",
    api_key="sk-your-openai-key",
    categories=["safety", "bias"]
)

# Custom endpoint - your own API
evaluation = client.evaluations.create_custom_endpoint(
    api_endpoint="https://api.yourcompany.com/v1",
    api_key="your-api-key",
    model_identifier="custom-model-v1"
)

Managing Evaluations

# List all evaluations
evaluations = client.evaluations.list()

# Filter by status
completed = client.evaluations.list(status="completed")

# Get detailed results
evaluation = client.evaluations.get(evaluation_id)
if evaluation.status == "completed":
    print(f"Overall Score: {evaluation.overall_score}")
    for score in evaluation.scores:
        print(f"{score.category}: {score.score:.2f}")

# Quick status check
status = client.evaluations.get_status(evaluation_id)
print(f"Progress: {status['completion_percentage']}%")

Configuration

Discover available options for evaluations:

# Get configuration options
config = client.config.get()

print("Available application types:")
for app_type in config.application_types:
    print(f"  {app_type['id']}: {app_type['name']}")

print("Available categories:")
for category in config.categories:
    print(f"  {category}")

print(f"Credits per category: {config.credits_per_category}")

Credits Management

Monitor your API key usage:

# Check credit balance
credits = client.credits.get_balance()

print(f"API Key: {credits.api_key_name}")
print(f"Credits Used: {credits.credits_used}")
print(f"Credits Remaining: {credits.credits_remaining}")
print(f"Credit Limit: {credits.credit_limit}")
print(f"Status: {credits.status}")

Error Handling

The SDK provides specific exceptions for different error types:

import trustmodel
from trustmodel import (
    AuthenticationError,
    ConnectionValidationError,
    InsufficientCreditsError,
    RateLimitError,
    ValidationError,
    APIError
)

try:
    client = trustmodel.TrustModelClient(api_key="tm-your-key")
    evaluation = client.evaluations.create(
        model_identifier="gpt-4",
        vendor_identifier="openai",
        api_key="sk-your-openai-key"  # BYOK
    )
except AuthenticationError:
    print("Invalid TrustModel API key")
except ConnectionValidationError as e:
    # BYOK or custom endpoint validation failed
    print(f"Vendor API key validation failed: {e.message}")
    if e.validation_details:
        status_code = e.validation_details.get("status_code")
        if status_code == 401:
            print("Check your vendor API key is valid and not expired")
        elif status_code == 404:
            print("Model not found - check the model identifier")
except InsufficientCreditsError as e:
    print(f"Need {e.credits_required} credits, but only {e.credits_remaining} remaining")
except RateLimitError:
    print("Rate limit exceeded, please wait")
except ValidationError as e:
    print(f"Invalid input: {e}")
except APIError as e:
    print(f"API error: {e.message} (status: {e.status_code})")

Exception Reference

Exception	When Raised
AuthenticationError	Invalid TrustModel API key
ConnectionValidationError	BYOK or custom endpoint API key validation failed
InsufficientCreditsError	Not enough credits for the evaluation
RateLimitError	Too many requests, need to wait
ValidationError	Invalid input parameters
ModelNotFoundError	Requested model doesn't exist
EvaluationNotFoundError	Requested evaluation doesn't exist
APIError	General API error (base class)

Webhook Notifications

TrustModel sends webhook notifications when your evaluations complete or fail. Configure your webhook endpoint in the TrustModel Dashboard to receive these events.

Success Event: sdk_report_evaluation_success

Sent when an evaluation completes successfully:

{
  "event_type": "sdk_report_evaluation_success",
  "timestamp": "2026-01-21T13:41:44.253319+00:00",
  "evaluation_run_id": 82,
  "model_identifier": "gpt-4",
  "status": "completed",
  "completion_percentage": 100,
  "overall_score": 65,
  "category_scores": [
    {
      "category_name": "Accuracy",
      "category_score": 100.0,
      "subcategories": [
        {
          "subcategory_name": "Citation & Source Accuracy",
          "subcategory_score": 100.0
        }
      ]
    }
  ]
}

Failure Event: sdk_report_evaluation_failed

Sent when an evaluation fails:

{
  "event_type": "sdk_report_evaluation_failed",
  "timestamp": "2026-01-21T12:38:18.349320+00:00",
  "evaluation_run_id": 78,
  "model_identifier": "gpt-4",
  "failed_phase": "evaluation",
  "failed_at": "2026-01-21T12:38:18.341673+00:00"
}

Webhook Event Fields

Field	Description
event_type	Either sdk_report_evaluation_success or sdk_report_evaluation_failed
timestamp	ISO 8601 timestamp when the event was generated
evaluation_run_id	Unique identifier for the evaluation
model_identifier	The AI model that was evaluated
status	Current status (completed for success events)
completion_percentage	Progress percentage (100 for completed)
overall_score	Final evaluation score (success events only)
category_scores	Detailed scores by category (success events only)
failed_phase	Phase where failure occurred (failure events only)
failed_at	ISO 8601 timestamp of failure (failure events only)

Advanced Usage

Context Manager

Use the client as a context manager for automatic cleanup:

with trustmodel.TrustModelClient(api_key="tm-your-key") as client:
    evaluation = client.evaluations.create(
        model_identifier="gpt-4",
        vendor_identifier="openai"
    )
    # Client automatically closed when exiting context

Custom Configuration

# Custom base URL and timeouts
client = trustmodel.TrustModelClient(
    api_key="tm-your-key",
    base_url="https://api-staging.trustmodel.ai",  # Use staging environment
    timeout=120,  # 2 minute timeout
    max_retries=5  # More aggressive retrying
)

Detailed Evaluation Configuration

evaluation = client.evaluations.create(
    model_identifier="gpt-4",
    vendor_identifier="openai",
    categories=["safety", "bias", "performance"],

    # Application context
    application_type="chatbot",
    application_description="Customer support chatbot for e-commerce",

    # User personas
    user_personas=["external-customer", "technical-user"],

    # Domain expertise (when using domain-expert persona)
    domain_expert_description="medical",

    # Custom naming
    model_config_name="GPT-4 Production Eval 2024-01"
)

Framework Integration

FastAPI

from fastapi import FastAPI, HTTPException
import trustmodel

app = FastAPI()
client = trustmodel.TrustModelClient(api_key="tm-your-key")

@app.post("/evaluate")
async def create_evaluation(model: str, vendor: str):
    try:
        evaluation = client.evaluations.create(
            model_identifier=model,
            vendor_identifier=vendor
        )
        return {"evaluation_id": evaluation.id, "status": evaluation.status}
    except trustmodel.InsufficientCreditsError:
        raise HTTPException(status_code=402, detail="Insufficient credits")

Django

# views.py
from django.http import JsonResponse
import trustmodel

def evaluate_model(request):
    client = trustmodel.TrustModelClient(api_key=settings.TRUSTMODEL_API_KEY)

    evaluation = client.evaluations.create(
        model_identifier=request.POST["model"],
        vendor_identifier=request.POST["vendor"]
    )

    return JsonResponse({
        "evaluation_id": evaluation.id,
        "status": evaluation.status
    })

Flask

from flask import Flask, request, jsonify
import trustmodel

app = Flask(__name__)
client = trustmodel.TrustModelClient(api_key="tm-your-key")

@app.route("/evaluate", methods=["POST"])
def evaluate():
    data = request.get_json()

    evaluation = client.evaluations.create(
        model_identifier=data["model"],
        vendor_identifier=data["vendor"]
    )

    return jsonify({
        "evaluation_id": evaluation.id,
        "status": evaluation.status
    })

Requirements

• Python 3.9 or higher
• requests >= 2.25.0
• pydantic >= 2.0.0

Support

• 📚 Documentation
• 🐛 Issues
• 💬 Support

License

This project is licensed under a proprietary license - see the LICENSE file for details.

Important: This SDK is provided exclusively for use with TrustModel's official API services. Modification, redistribution, or reverse engineering is prohibited.