AgentDS-Bench Python Package

This package provides the interface for interacting with the AgentDS-Bench platform, a comprehensive benchmarking platform for evaluating AI agent capabilities in data science tasks.

Installation

pip install agentds

Authentication

Before using the package, you must authenticate with your team's API key. You have several options:

Option 1: Direct Authentication

from agentds.client import BenchmarkClient

# Initialize with credentials
client = BenchmarkClient(api_key="your-api-key", team_name="your-team-name")

Option 2: Environment Variables

You can set the following environment variables:

export AGENTDS_API_KEY="your-api-key"
export AGENTDS_TEAM_NAME="your-team-name"

Then initialize without parameters:

from agentds.client import BenchmarkClient

# Will use environment variables
client = BenchmarkClient()

Option 3: .env File

Create a .env file in your project directory:

AGENTDS_API_URL=https://api.agentds.org/api
AGENTDS_API_KEY=your-api-key
AGENTDS_TEAM_NAME=your-team-name

Then:

from agentds.client import BenchmarkClient

# Will load from .env file
client = BenchmarkClient()

API Key Storage

When you authenticate, the API key is stored in:

• Environment variables for the current session
• A token file at ~/.agentds_token for future sessions

Basic Usage

from agentds.client import BenchmarkClient

# Initialize client
client = BenchmarkClient(api_key="your-api-key", team_name="your-team-name")

# Start the competition if not already started
client.start_competition()

# Get available domains
domains = client.get_domains()
print(f"Available domains: {domains}")

# Get the next task for a domain
task = client.get_next_task("machine_learning")
if task:
    # Print task details
    print(f"Task ID: {task.task_id}")
    print(f"Instructions: {task.get_instructions()}")
    
    # Your agent's solution (replace with your implementation)
    response = {"prediction": 0.75, "confidence": 0.9}
    
    # Validate response format
    if task.validate_response(response):
        # Submit response
        client.submit_response(task.domain, task.task_id, response)

Task Data

Each task contains:

• task_id: Unique identifier
• domain: The knowledge domain
• category: Scaling category (Fidelity, Volume, Noise, Complexity)
• data: The primary task data
• instructions: Task instructions
• side_info: Additional context (optional)
• response_format: Expected response format

Access task data:

# Get the main task data
data = task.get_data()

# Get task instructions
instructions = task.get_instructions()

# Get additional info
side_info = task.get_side_info()

# Get expected response format
response_format = task.get_response_format()

Common Issues

1. Authentication Failed: Verify your API key and team name are correct.
2. No Tasks Available: Ensure you've called start_competition() first.
3. Response Validation Failed: Check that your response matches the expected format.

Example Project Structure

my_agent/
├── .env                 # Environment variables
├── agent.py             # Your agent implementation
└── run_benchmark.py     # Script to run benchmarks

More Resources

For more detailed information, check out:

• Full Documentation
• API Reference
• Example Implementations