cat-cafe-client 0.10.0

Python client for CAT Cafe - Continuous Alignment Testing platform for LLM observability

CAT Cafe SDK

Python SDK for CAT Cafe - Continuous Alignment Testing platform for LLM observability.

Installation

pip install cat-cafe-client

Compatibility: the client version tracks the CAT Cafe server API it targets. cat-cafe-client 0.10.0 targets the cat-cafe server 0.10.0 API.

Quick Start

from cat.cafe.client import CATCafeClient, DatasetImport, DatasetExample

# Initialize the client
client = CATCafeClient(base_url="http://localhost:8000")

# Create a dataset with examples in one call
dataset = DatasetImport(
    name="My Test Dataset",
    description="Sample dataset for testing",
    examples=[
        DatasetExample(
            input={"messages": [{"role": "user", "content": "What's the weather?"}]},
            expected={"messages": [{"role": "assistant", "content": "Weather info"}]},
            metadata={"tags": ["weather"]},
        )
    ],
)
result = client.import_dataset(dataset)
dataset_id = result["dataset"]["id"]

From here, run your AI system against the dataset and stream results to an experiment -- see "Manual Experiment Control" below for the full flow.

This SDK is the core HTTP client for CAT Cafe's dataset, experiment, and trace-scoring APIs. It intentionally does not include a task/evaluator runner -- for the higher-level workflow of defining a task function, evaluators, and running them against a dataset, use cat-experiments instead:

pip install "cat-experiments[cat-cafe]"

cat-experiments run experiment.py --dataset data.jsonl --storage cat-cafe

Core Classes

CATCafeClient

The main client for interacting with CAT Cafe:

client = CATCafeClient(base_url="http://localhost:8000")

Dataset Models

• DatasetImport: For creating new datasets with examples
• DatasetExample: Individual examples in a dataset
• Dataset: Structured dataset object returned from API
• Example: Individual example object returned from API

Experiment Models

• Experiment: Experiment configuration
• ExperimentResult: Results from running experiments

Key Methods

Dataset Operations

# Import a complete dataset
result = client.import_dataset(dataset_import)

# Add examples to an existing dataset (single or batch)
example_id = client.add_dataset_example(dataset_id, example)
client.add_dataset_examples(dataset_id, [example_a, example_b])

# Fetch dataset as structured object
dataset = client.fetch_dataset(dataset_id)

# Find dataset by name
dataset = client.fetch_dataset_by_name("My Dataset")

# List all datasets
datasets = client.list_datasets()

Experiment Operations

# Start an experiment tied to a dataset
experiment_id = client.start_experiment(experiment_config)

# Stream runs as you produce them
client.create_run(experiment_id, {...})
client.append_evaluation(experiment_id, run_id, {...})

# Mark it done
client.complete_experiment(experiment_id)

# Compare two experiments on the same dataset
comparison = client.compare_experiments(experiment_a_id, experiment_b_id)

Manual Experiment Control

from cat.cafe.client import CATCafeClient, Experiment

client = CATCafeClient(base_url="http://localhost:8000")

# Create experiment configuration
experiment_config = Experiment(
    name="Manual Experiment",
    description="Step-by-step experiment",
    dataset_id=dataset_id,
    tags=["manual", "testing"],
)

# Start experiment
experiment_id = client.start_experiment(experiment_config)

# Run your tests and stream results
dataset = client.fetch_dataset(dataset_id)

for example in dataset.examples:
    output = call_my_ai_system(example.input)  # however you invoke your system under test

    client.create_run(
        experiment_id,
        {
            "run_id": f"run-{example.id}",
            "example_id": example.id,
            "input_data": {"input": example.input},
            "output": dict(example.expected) if isinstance(example.expected, dict) else example.expected,
            "actual_output": output,
        },
    )
    client.append_evaluation(
        experiment_id,
        f"run-{example.id}",
        {"evaluator_name": "manual_score", "score": 0.8},
    )

# Complete experiment
client.complete_experiment(experiment_id, {"total_examples": len(dataset.examples)})

Requirements

• Python 3.10+
• httpx
• CAT Cafe server running (default: http://localhost:8000)

License

Apache License 2.0 — see the LICENSE file in the cat-cafe repository for details.