playgent 0.1.0

SDK for tracking and testing OpenAI API calls

Playgent

Playgent is a Python SDK for tracking, testing, and evaluating OpenAI API calls. It provides seamless integration with OpenAI's Python client, allowing you to record API interactions, replay them for testing, and evaluate model outputs systematically.

Features

• Session Tracking: Automatically track and record OpenAI API calls within sessions
• Replay Testing: Replay recorded API calls to test different scenarios
• Evaluation Framework: Evaluate model outputs against expected results
• Drop-in Replacement: Works as a drop-in replacement for the OpenAI client
• Async Support: Full support for both synchronous and asynchronous operations
• Decorator Pattern: Simple @record decorator for automatic session management

Installation

Install Playgent using pip:

pip install playgent

Or using uv:

uv add playgent

Quick Start

Basic Usage

from playgent import init, create_session, record

# Initialize Playgent with your configuration
init(api_key="your-playgent-api-key", server_url="https://your-server.com")

# Create a session
session_id = create_session(
    person_id="user-123",
    endpoint="chat-endpoint",
    metadata={"purpose": "testing"}
)

# Your OpenAI code here - all calls will be tracked automatically

Using the Drop-in OpenAI Client

Replace your OpenAI import with Playgent's version:

# Instead of: from openai import OpenAI
from playgent.openai import OpenAI

client = OpenAI(api_key="your-openai-api-key")

# Use as normal - all calls are automatically tracked
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}]
)

Using the @record Decorator

The simplest way to track OpenAI calls:

from playgent import record
from openai import OpenAI

@record(
    person_id="user-123",
    endpoint="my-endpoint"
)
def my_function():
    client = OpenAI()
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": "Hello!"}]
    )
    return response

# All OpenAI calls within this function are automatically tracked
result = my_function()

Async Support

from playgent import record
from openai import AsyncOpenAI
import asyncio

@record(
    person_id="user-123",
    endpoint="async-endpoint"
)
async def my_async_function():
    client = AsyncOpenAI()
    response = await client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": "Hello!"}]
    )
    return response

# Run async function
asyncio.run(my_async_function())

Replay Testing

from playgent import replay_test, get_session_events

# Get events from a session
events = get_session_events(session_id="your-session-id")

# Create a test case
test_case = {
    "name": "Test Chat Response",
    "session_id": "your-session-id",
    "events": events
}

# Replay the test
result = replay_test(test_case)
print(f"Test passed: {result['passed']}")

Evaluation

from playgent import evaluate

# Evaluate model outputs
evaluation = evaluate(
    session_id="your-session-id",
    criteria={
        "accuracy": lambda output: len(output) > 10,
        "relevance": lambda output: "expected" in output.lower()
    }
)

print(f"Evaluation results: {evaluation}")

Configuration

Playgent can be configured through environment variables or programmatically:

Environment Variables

export PLAYGENT_API_KEY="your-api-key"
export PLAYGENT_SERVER_URL="https://your-server.com"
export PLAYGENT_PERSON_ID="default-user"
export PLAYGENT_ENDPOINT="default-endpoint"

Programmatic Configuration

from playgent import init

init(
    api_key="your-api-key",
    server_url="https://your-server.com",
    batch_size=100,  # Number of events to batch before sending
)

Advanced Usage

Session Management

from playgent import create_session, get_session, session

# Create a session with metadata
session_id = create_session(
    person_id="user-123",
    endpoint="advanced-endpoint",
    metadata={
        "environment": "production",
        "version": "1.0.0",
        "feature_flags": ["new_model", "streaming"]
    }
)

# Get session details
session_info = get_session(session_id)
print(f"Session: {session_info}")

# Use session context manager
with session(person_id="user-123", endpoint="context-endpoint"):
    # All OpenAI calls in this block are tracked
    pass

Custom Event Tracking

from playgent import record
from playgent.types import EndpointEvent

# Track custom events alongside OpenAI calls
@record(person_id="user-123", endpoint="custom-endpoint")
def process_with_custom_events():
    # Your OpenAI calls here

    # Create custom event
    event = EndpointEvent(
        type="custom_metric",
        data={"metric": "response_time", "value": 0.125}
    )

    # Event will be included in session
    return result

API Reference

Core Functions

• init(api_key, server_url, batch_size): Initialize Playgent
• create_session(person_id, endpoint, metadata): Create a new tracking session
• get_session(session_id): Get session details
• get_session_events(session_id): Get all events from a session
• replay_test(test_case): Replay a test case
• evaluate(session_id, criteria): Evaluate session outputs
• reset(): Reset Playgent state

Decorators

• @record(person_id, endpoint, metadata): Decorator for automatic session management

Types

• Session: Session data model
• EndpointEvent: Event data model
• TestCase: Test case definition
• EvaluationResult: Evaluation result data

Development

Setting Up Development Environment

# Clone the repository
git clone https://github.com/yourusername/playgent-py.git
cd playgent-py

# Install development dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Format code
black src/
ruff check src/

# Type checking
mypy src/

Building and Publishing

# Build the package
python -m build

# Test on TestPyPI first
python -m twine upload --repository testpypi dist/*

# Publish to PyPI
python -m twine upload dist/*

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

For support, please open an issue on the GitHub repository.

Acknowledgments

• Built for seamless integration with OpenAI's Python client
• Inspired by modern testing and observability practices
• Designed for production-grade API tracking and testing