modelscout-sdk 0.1.0

Python SDK for ModelScout - LLM Benchmarking and Evaluation

ModelScout Python SDK

Find the best LLM for your product. Run benchmarks across multiple models on your own data to see which performs best for quality, cost, and latency.

Installation

pip install modelscout-sdk

Quick Start

from modelscout import Benchmark, ModelConfig

# Set MODELSCOUT_API_KEY in your environment, or pass api_key="ms_..."
results = Benchmark().run(
    pack="trial",  # free trial, or "pb_..." from dashboard purchase
    prompts=["Write a SQL query to find active users", "Explain quantum computing"],
    models=[
        ModelConfig(provider="openai", model="gpt-5-mini"),
        ModelConfig(provider="anthropic", model="claude-haiku-4-5-20251001"),
        ModelConfig(provider="deepseek", model="deepseek-v3.2"),
    ],
)

print(results.best_model_for("quality"))  # Best quality model
print(results.best_model_for("cost"))     # Cheapest model

Features

Benchmarking

Compare LLMs side-by-side on your evaluation data. Get quality scores, cost analysis, latency metrics, and statistical significance.

Data Generation

Need synthetic test data? Generate evaluation datasets from the dashboard — describe your use case and get representative prompts in minutes.

Dataset Upload

Upload your own evaluation data:

dataset_id = benchmark.upload_dataset(
    name="My Test Data",
    samples=[
        {"input": "What is machine learning?"},
        {"input": "Explain neural networks"},
    ],
)

Agentic Evaluation

Test tool-calling capabilities with multi-turn evaluation (SDK-only):

from modelscout import Benchmark, ModelConfig, AgenticConfig, ToolDefinition

def my_search_function(query: str) -> str:
    return f"Results for: {query}"

config = AgenticConfig(tools=[
    ToolDefinition(
        name="search",
        description="Search the web",
        parameters={"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]},
        implementation=my_search_function,
    )
])

results = Benchmark().run(
    name="Agent Eval",
    pack="pb_...",
    prompts=["Find information about quantum computing"],
    models=[
        ModelConfig(provider="openai", model="gpt-5-mini"),
        ModelConfig(provider="anthropic", model="claude-haiku-4-5-20251001"),
    ],
    agentic_config=config,
)

Supported Models

23 models across 8 providers:

Provider	Models
OpenAI	gpt-5.4, gpt-5.4-mini, gpt-5.4-nano, gpt-5-mini, gpt-5-nano
Anthropic	claude-opus-4-6, claude-sonnet-4-6, claude-haiku-4-5-20251001
Google	gemini-3.1-pro, gemini-3-flash, gemini-3.1-flash-lite, gemini-2.5-flash-lite
DeepSeek	deepseek-v3.2, deepseek-v3.2-speciale, deepseek-r1
Qwen	qwen3.5-397b-a17b, qwen3.5-flash-02-23, qwen3-235b-a22b
Meta	llama-4-maverick, llama-4-scout
Mistral	mistral-large-2512, mistral-small-2603
xAI	grok-4, grok-4.1-fast

Pricing

Free trial: Every new organization gets one free benchmark (10 samples, 2 standard models).

Pay-as-you-go: Purchase benchmarks from the dashboard. Price depends on selected models, sample count, and judge tier. Starting from $4.99.

Documentation

Full documentation: modelscout.co/docs/sdk

---

License

Proprietary. See LICENSE for details.