agentlab-py 0.4.0

Python client library for the AgentLab evaluation platform using Connect RPC

AgentLab Python Client

A Python client library for the AgentLab evaluation platform using Connect RPC. This library provides a simple and intuitive interface for running AI agent evaluations, managing evaluators, and accessing evaluation results.

🚀 Quick Start

pip install agentlab-py

Set your API token as an environment variable:

export AGENTLAB_API_TOKEN=your-api-token-here

from agentlab import AgentLabClient, CreateEvaluationOptions

# Initialize the client (automatically loads AGENTLAB_API_TOKEN from environment)
client = AgentLabClient()

# Run an evaluation
evaluation = client.run_evaluation(CreateEvaluationOptions(
    agent_name='my-agent',
    agent_version='1.0.0',
    evaluator_names=['correctness-v1'],
    user_question='What is the capital of France?',
    agent_answer='The capital of France is Paris.',
    ground_truth='Paris is the capital of France',
    metadata={'confidence': 0.95}  # Optional metadata for tracking
))

print(f"Evaluation completed: {evaluation.name}")

Retrieving Results

# Get evaluation run details
evaluation_run = client.get_evaluation_run('evaluation-run-id')

# Get structured results with parsed JSON
result_data = client.get_evaluation_result('evaluation-run-id')
print(result_data['results'])  # Parsed evaluator outputs

# Access raw evaluator results
for evaluator_name, result in evaluation_run.evaluator_results.items():
    print(f"{evaluator_name}: {result.output}")

Listing Evaluation Runs

# List recent evaluation runs
runs_response = client.list_evaluation_runs('project-123')
for run in runs_response.evaluation_runs:
    print(f"Run: {run.name} - Question: {run.user_question}")

🔧 Development

Setting up the development environment

# Clone the repository
git clone https://github.com/VectorLabsCZ/agentlab-py.git
cd agentlab-py

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install development dependencies
pip install -r requirements-dev.txt

# Install the package in development mode
pip install -e .

Running Examples

# Basic usage example
python examples/basic_usage.py

# Async usage example
python examples/async_usage.py

Running Tests

# Run tests
pytest

# Run tests with coverage
pytest --cov=agentlab

# Run type checking
mypy agentlab/

📦 Building and Publishing

Building the package

# Install build tools
pip install build

# Build the package
python -m build

Publishing to PyPI

# Install twine
pip install twine

# Upload to PyPI
twine upload dist/*

🌟 Examples

Complete Evaluation Workflow

import json
from agentlab import AgentLabClient, CreateEvaluationOptions

def main():
    # Initialize client
    client = AgentLabClient()
        
    try:
        # 1. List available evaluators
        print("📋 Available evaluators:")
        evaluators = client.list_evaluators()
        for evaluator in evaluators.evaluators[:3]:  # Show first 3
            print(f"  - {evaluator.name}: {evaluator.display_name}")
        
        # 2. Run evaluation
        print("\n🚀 Running evaluation...")
        evaluation = client.run_evaluation(CreateEvaluationOptions(
            agent_name='demo-agent',
            agent_version='1.0.0',
            evaluator_names=['correctness-v1'],
            user_question='What is the square root of 16?',
            agent_answer='The square root of 16 is 4.',
            ground_truth='4',
            metadata={'confidence': 1.0}  # Additional context/scores
        ))
        
        # 3. Get results
        print(f"\n✅ Evaluation completed: {evaluation.name}")
        result_data = client.get_evaluation_result(evaluation.name)
        
        print("\n📊 Results:")
        print(json.dumps(result_data['results'], indent=2))
        
        # 4. List recent runs
        print("\n📈 Recent evaluation runs:")
        runs = client.list_evaluation_runs()
        for run in runs.evaluation_runs[:3]:  # Show first 3
            print(f"  - {run.name}: {run.user_question[:50]}...")
            
    except Exception as e:
        print(f"❌ Error: {e}")

if __name__ == '__main__':
    main()

Error Handling

from agentlab import AgentLabClient, AgentLabClientOptions, AgentLabError, AuthenticationError, APIError

try:
    client = AgentLabClient(AgentLabClientOptions(api_token='invalid-token'))
    evaluation = client.run_evaluation(options)
    
except AuthenticationError as e:
    print(f"Authentication failed: {e}")
    
except APIError as e:
    print(f"API error (status {e.status_code}): {e}")
    
except AgentLabError as e:
    print(f"AgentLab error: {e}")
    
except Exception as e:
    print(f"Unexpected error: {e}")

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Issues and Feature Requests

• 🐛 Report a bug
• 💡 Request a feature

🔗 Links

• AgentLab Platform
• Documentation
• JavaScript SDK

🏢 About VectorLabs

AgentLab is developed by VectorLabs, a company focused on advancing AI agent evaluation and development tools.

---

Made with ❤️ by the VectorLabs team