trainloop-evals-sdk 0.2.0

TrainLoop Evaluations SDK for data collection and evaluation

TrainLoop Evals SDK

A lightweight Python SDK for collecting, submitting, and evaluating LLM responses with TrainLoop.

Installation

pip install trainloop-evals-sdk

Usage

Data Collection

from trainloop_evals import collect

# Set this up during the initialization of your application
# to collect data from all LLM calls
collect()

from anthropic import Anthropic
from openai import OpenAI



# Annotate llm calls with the trainloop_tag parameter
# the sdk will intelligently remove the parameter and collect information about the llm call
import openai

openai.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "2+2"}],
    trainloop_tag="math.addition"
)

IMPORTANT: The collect() function must be called in your entrypoint BEFORE you import any LLM clients. Otherwise their requests will not be patched!

Configuration

The SDK can be configured using environment variables:

• TRAINLOOP_DATA_FOLDER: Directory where the registry file will be saved. If this is not set, collect() becomes a no-op.

Advanced Usage

See the documentation for complete API reference.