modelscout-sdk 0.2.0

Python SDK for ModelScout - LLM Benchmarking and Evaluation

ModelScout Python SDK

Find the best LLM for your product. Run benchmarks across multiple models on your own data to see which performs best for quality, cost, and latency.

Installation

pip install modelscout-sdk

Quick Start

from modelscout import Benchmark

# Set MODELSCOUT_API_KEY in your environment, or pass api_key="ms_..."
# Models are selected at checkout and locked to your purchase
results = Benchmark().run(
    purchased_benchmark_id="pb_...",  # from dashboard checkout
    prompts=["Write a SQL query to find active users", "Explain quantum computing"],
)

print(results.best_model_for("quality"))  # Best quality model
print(results.best_model_for("cost"))     # Cheapest model

Features

Benchmarking

Compare LLMs side-by-side on your evaluation data. Get quality scores, cost analysis, latency metrics, and statistical significance.

Data Generation

Need synthetic test data? Generate evaluation datasets from the dashboard — describe your use case and get representative prompts in minutes.

Dataset Upload

Upload your own evaluation data:

dataset_id = benchmark.upload_dataset(
    name="My Test Data",
    samples=[
        {"input": "What is machine learning?"},
        {"input": "Explain neural networks"},
    ],
)

Agentic Evaluation

Test tool-calling capabilities with multi-turn evaluation (SDK-only):

from modelscout import Benchmark, AgenticConfig, ToolDefinition

def my_search_function(query: str) -> str:
    return f"Results for: {query}"

config = AgenticConfig(tools=[
    ToolDefinition(
        name="search",
        description="Search the web",
        parameters={"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]},
        implementation=my_search_function,
    )
])

# Models are locked to your purchase — selected at checkout
results = Benchmark().run(
    name="Agent Eval",
    purchased_benchmark_id="pb_...",
    prompts=["Find information about quantum computing"],
    agentic_config=config,
)

Execution Graders

Pass a grader= to Benchmark.run() to add a deterministic pass/fail layer that runs on your machine, against your ground truth. LLM judges score style well but can miss correctness on structured outputs like SQL, JSON, or code — graders fill that gap. The verdict is both passed to the judge as primary evidence of correctness and surfaced independently as execution_pass_rate_by_model.

Pre-built graders: SQLGrader, JSONSchemaGrader, NumericGrader. Subclass Grader for anything else.

from modelscout import Benchmark
from modelscout.graders import SQLGrader

grader = SQLGrader(db_path="./warehouse.sqlite")

result = Benchmark(api_key="ms_...").run(
    name="SQL Benchmark",
    purchased_benchmark_id="pb_...",
    samples=[
        {
            "input": "Which products sold the most last quarter?",
            # expected_output is the canonical row set as JSON
            "expected_output": '[["Widget", 1200], ["Gadget", 900]]',
        },
    ],
    grader=grader,
)

print(result.execution_pass_rate_by_model)
# {'anthropic/claude-opus-4-7': 48.0, 'openai/gpt-5.4': 28.0}

Full guide: Graders.

Supported Models

30 models across 10 providers:

Provider	Models
OpenAI	gpt-5.4, gpt-5.4-mini, gpt-5.4-nano, gpt-5-mini, gpt-5-nano, gpt-oss-120b, gpt-oss-20b
Anthropic	claude-opus-4-7, claude-opus-4-6, claude-sonnet-4-6, claude-haiku-4-5-20251001
Google	gemini-3.1-pro, gemini-3-flash, gemini-3.1-flash-lite, gemini-2.5-flash-lite, gemma-3-27b-it
DeepSeek	deepseek-v3.2, deepseek-v3.2-speciale, deepseek-r1
Qwen	qwen3.5-397b-a17b, qwen3.5-flash-02-23, qwen3-235b-a22b
Meta	llama-4-maverick, llama-4-scout
Mistral	mistral-large-2512, mistral-small-2603
xAI	grok-4, grok-4.1-fast
Zhipu	glm-5, glm-5-turbo, glm-5.1
Moonshot	kimi-k2.5

Pricing

Pay-as-you-go: Purchase benchmarks from the dashboard. Price depends on selected models, sample count, and judge tier. Starting from ~$4.50.

Launch Discount: 10% off all benchmarks during our launch period. Applied automatically at checkout.

Documentation

Full documentation: modelscout.co/docs/sdk

---

License

Proprietary. See LICENSE for details.