Tiny Agents. Total Visibility.

Smol Agents. Smart Metrics.

Documentation | Quickstart | PyPI | Issues

smoltrace

SMOLTRACE is a benchmarking and evaluation framework for Smolagents, Hugging Face's lightweight agent library. It runs ToolCallingAgent and CodeAgent against curated or custom task datasets, auto-instruments everything with OpenTelemetry, and publishes results, traces, metrics, and a leaderboard — with zero configuration.

Get Started in 30 Seconds

pip install smoltrace
export HF_TOKEN=hf_your_token
export OPENAI_API_KEY=sk-your_key   # or MISTRAL_API_KEY, ANTHROPIC_API_KEY, ...

smoltrace-eval --model openai/gpt-4.1-nano --provider litellm --agent-type both --enable-otel

That's it. One command evaluates both agent types and automatically creates 4 datasets on the Hugging Face Hub:

• {username}/smoltrace-results-{timestamp} — per-test-case outcomes
• {username}/smoltrace-traces-{timestamp} — OpenTelemetry spans
• {username}/smoltrace-metrics-{timestamp} — GPU / CO2 / cost time-series
• {username}/smoltrace-leaderboard — aggregate stats, updated every run

No repo names to configure — everything is derived from your HF_TOKEN.

Key Features

• Zero configuration — only HF_TOKEN required; dataset names are auto-generated from your username
• Multi-provider — LiteLLM (OpenAI, Anthropic, Mistral, Groq, ...), HF Inference API, local Transformers models on GPU, and Ollama
• OTEL native — auto-instrumented via genai-otel-instrument: spans, token counts, cost tracking, CO2 emissions
• GPU metrics — utilization, memory, temperature, and power tracked by default for local models, flattened for dashboards
• 20+ agent tools — web search, webpage reading, file system, text processing (grep/sed/sort), and process/system tools (ps/curl/ping), with working-directory sandboxing
• 3 benchmark datasets — quick tasks (13), comprehensive benchmark (132: GAIA/Math/SimpleQA), and an ops/SRE benchmark (24)
• Flexible output — push to HF Hub, save local JSON, or export to OpenSearch with typed mappings
• Parallel execution — --parallel-workers 8 for 10-50x faster API-model evaluations
• Dataset lifecycle — smoltrace-copy-datasets to bootstrap, smoltrace-cleanup to safely prune old runs
• HF Jobs ready — run on Hugging Face cloud hardware (CPU to A100) with pay-as-you-go billing

Installation

pip install smoltrace                # core
pip install smoltrace[gpu]           # + GPU metrics for local models
pip install smoltrace[opensearch]    # + OpenSearch export

Requires Python 3.11+. For development installs and full requirements, see the installation guide.

Usage

CLI

# API model via LiteLLM (default provider)
smoltrace-eval --model openai/gpt-4 --provider litellm --agent-type tool

# Local GPU model via Transformers (GPU metrics on by default)
smoltrace-eval --model meta-llama/Llama-3.1-8B --provider transformers --agent-type both

# Local model via Ollama
smoltrace-eval --model qwen2.5-coder:3b --provider ollama --agent-type tool --enable-otel

# HF Inference API
smoltrace-eval --model meta-llama/Llama-3.1-70B-Instruct --provider inference --agent-type both

# Enable agent tools, filter by difficulty, run in parallel
smoltrace-eval \
  --model openai/gpt-4.1-nano \
  --provider litellm \
  --enable-tools visit_webpage read_file grep \
  --working-directory ./workspace \
  --difficulty easy \
  --parallel-workers 8 \
  --enable-otel

Other useful flags: --dataset-name (custom task dataset), --model-args temperature=0.7 seed=42, --prompt-yml (custom prompts: a smolagents prompt-template file, e.g. prompt_template.tool.yaml, merged over the defaults; a lone plain system_prompt is prepended to the default; a missing or unparseable file stops the run rather than falling back to the default prompts), --mcp-server-url (MCP tools), --private (private datasets), --output-format hub|json|opensearch.

Full flag reference: CLI documentation.

Python API

import os
from smoltrace.core import run_evaluation

all_results, trace_data, metric_data, dataset_used, run_id = run_evaluation(
    model="openai/gpt-4",
    provider="litellm",
    agent_type="both",
    enable_otel=True,
    hf_token=os.getenv("HF_TOKEN"),
)

Lower-level building blocks (generate_dataset_names, push_results_to_hf, compute_leaderboard_row, update_leaderboard, ...) are documented in the Python API reference.

Benchmark Datasets

Copy them to your own account with smoltrace-copy-datasets, or bring your own tasks as a HF dataset or local JSON. Schemas, the ops-benchmark sample-data setup, and custom task creation are covered in the datasets guide.

Community task datasets: the TraceMind-AI collection bundles 40+ ready-to-run task datasets spanning real-world domains — travel, e-commerce, healthcare, finance, DevOps/SRE, AIOps, cybersecurity, and more — for evaluating agents on domain-specific workloads. Point --dataset-name at any of them, e.g. --dataset-name MCP-1st-Birthday/smoltrace-finance-tasks.

Output Formats

--output-format hub          # default: 4 datasets on the HF Hub
--output-format json         # 5 local JSON files (results, traces, metrics, leaderboard row, metadata)
--output-format opensearch   # 4 OpenSearch indexes with typed mappings and bulk indexing

Details and OpenSearch connection options: output formats guide.

Run on HuggingFace Jobs

Run evaluations on HF cloud hardware without a local GPU (Pro / Team / Enterprise):

hf jobs run \
  --flavor cpu-basic \
  -s HF_TOKEN=hf_your_token \
  -s OPENAI_API_KEY=your_key \
  python:3.12 \
  bash -c "pip install smoltrace ddgs && smoltrace-eval --model openai/gpt-4 --provider litellm --enable-otel"

GPU flavors (t4-small through a100-large), the Python API, and scheduled evaluations are covered in the HF Jobs guide.

Dataset Management

Each run creates 3 timestamped datasets, so periodic cleanup keeps your profile tidy:

smoltrace-cleanup --older-than 30d            # dry-run preview (default)
smoltrace-cleanup --keep-recent 5 --no-dry-run  # actually delete

Cleanup is dry-run by default, never touches your leaderboard, and permanently protects the benchmark/tasks datasets. See the dataset management guide.

Leaderboard

Every run appends aggregate stats (success rate, pass@1, avg steps, tokens, duration, cost, CO2, GPU utilization) to your personal leaderboard, with links back to the run's results, traces, and metrics datasets. SMOLTRACE computes pass@1 from the first recorded completion for each logical (agent_type, test_id) task and publishes the rule and denominator alongside the score. Community rankings live at huggingface/smolagents-leaderboard — contribute your runs! More in the leaderboard guide.

Documentation

• Quickstart — first evaluation in minutes
• Configuration — environment variables and options
• Running evaluations — providers, difficulty filters, parallelism
• Agent tools — all 20+ tools with security notes
• GPU metrics & cost — what's collected and how it's aggregated
• Changelog
• llms.txt — machine-readable docs index for LLM agents

Community

• Discord — chat with the community (shared with genai-otel-instrument)
• TraceVerse Community — an open org on the Hub for evaluation & observability datasets, spaces, and real-world MCP apps; join and share your own
• TraceMind-AI collection — 40+ domain-specific task datasets to evaluate against
• GitHub Issues — bug reports and feature requests

Contributing

Contributions welcome! See the contributing guide.

git clone https://github.com/Mandark-droid/SMOLTRACE.git && cd SMOLTRACE
pip install -e .[dev]
pytest

License

Apache-2.0. See LICENSE.

⭐ Star this repo to support Smolagents! Questions? Open an issue.