Tiny Agents. Total Visibility.

Smol Agents. Smart Metrics.

Documentation | Quickstart | PyPI | Issues

smoltrace

SMOLTRACE is a benchmarking and evaluation framework for Smolagents, Hugging Face's lightweight agent library. It runs ToolCallingAgent and CodeAgent against curated or custom task datasets, auto-instruments everything with OpenTelemetry, and publishes results, traces, metrics, and a leaderboard — with zero configuration.

Get Started in 30 Seconds

pip install smoltrace
export HF_TOKEN=hf_your_token
export OPENAI_API_KEY=sk-your_key   # or MISTRAL_API_KEY, ANTHROPIC_API_KEY, ...

smoltrace-eval --model openai/gpt-4.1-nano --provider litellm --agent-type both --enable-otel

That's it. One command evaluates both agent types and automatically creates 4 datasets on the Hugging Face Hub:

• {username}/smoltrace-results-{timestamp} — per-test-case outcomes
• {username}/smoltrace-traces-{timestamp} — OpenTelemetry spans
• {username}/smoltrace-metrics-{timestamp} — GPU / CO2 / cost time-series
• {username}/smoltrace-leaderboard — aggregate stats, updated every run

No repo names to configure — everything is derived from your HF_TOKEN.

Key Features

• Zero configuration — only HF_TOKEN required; dataset names are auto-generated from your username
• Multi-provider — LiteLLM (OpenAI, Anthropic, Mistral, Groq, ...), HF Inference API, local Transformers models on GPU, and Ollama
• OTEL native — auto-instrumented via genai-otel-instrument: spans, token counts, cost tracking, CO2 emissions
• GPU metrics — utilization, memory, temperature, and power tracked by default for local models, flattened for dashboards
• 20+ agent tools — web search, webpage reading, file system, text processing (grep/sed/sort), and process/system tools (ps/curl/ping), with working-directory sandboxing
• 3 benchmark datasets — quick tasks (13), comprehensive benchmark (132: GAIA/Math/SimpleQA), and an ops/SRE benchmark (24)
• Flexible output — push to HF Hub, save local JSON, or export to OpenSearch with typed mappings
• Parallel execution — --parallel-workers 8 for 10-50x faster API-model evaluations
• Dataset lifecycle — smoltrace-copy-datasets to bootstrap, smoltrace-cleanup to safely prune old runs
• HF Jobs ready — run on Hugging Face cloud hardware (CPU to A100) with pay-as-you-go billing

Installation

pip install smoltrace                # core
pip install smoltrace[gpu]           # + GPU metrics for local models
pip install smoltrace[opensearch]    # + OpenSearch export

Requires Python 3.10+. For development installs and full requirements, see the installation guide.

Usage

CLI

# API model via LiteLLM (default provider)
smoltrace-eval --model openai/gpt-4 --provider litellm --agent-type tool

# Local GPU model via Transformers (GPU metrics on by default)
smoltrace-eval --model meta-llama/Llama-3.1-8B --provider transformers --agent-type both

# Local model via Ollama
smoltrace-eval --model qwen2.5-coder:3b --provider ollama --agent-type tool --enable-otel

# HF Inference API
smoltrace-eval --model meta-llama/Llama-3.1-70B-Instruct --provider inference --agent-type both

# Enable agent tools, filter by difficulty, run in parallel
smoltrace-eval \
  --model openai/gpt-4.1-nano \
  --provider litellm \
  --enable-tools visit_webpage read_file grep \
  --working-directory ./workspace \
  --difficulty easy \
  --parallel-workers 8 \
  --enable-otel

Other useful flags: --dataset-name (custom task dataset), --model-args temperature=0.7 seed=42, --prompt-yml (custom prompts), --mcp-server-url (MCP tools), --private (private datasets), --output-format hub|json|opensearch.

Full flag reference: CLI documentation.

Python API

import os
from smoltrace.core import run_evaluation

all_results, trace_data, metric_data, dataset_used, run_id = run_evaluation(
    model="openai/gpt-4",
    provider="litellm",
    agent_type="both",
    enable_otel=True,
    hf_token=os.getenv("HF_TOKEN"),
)

Lower-level building blocks (generate_dataset_names, push_results_to_hf, compute_leaderboard_row, update_leaderboard, ...) are documented in the Python API reference.

Benchmark Datasets

Copy them to your own account with smoltrace-copy-datasets, or bring your own tasks as a HF dataset or local JSON. Schemas, the ops-benchmark sample-data setup, and custom task creation are covered in the datasets guide.

Community task datasets: the TraceMind-AI collection bundles 40+ ready-to-run task datasets spanning real-world domains — travel, e-commerce, healthcare, finance, DevOps/SRE, AIOps, cybersecurity, and more — for evaluating agents on domain-specific workloads. Point --dataset-name at any of them, e.g. --dataset-name MCP-1st-Birthday/smoltrace-finance-tasks.

Output Formats

--output-format hub          # default: 4 datasets on the HF Hub
--output-format json         # 5 local JSON files (results, traces, metrics, leaderboard row, metadata)
--output-format opensearch   # 4 OpenSearch indexes with typed mappings and bulk indexing

Details and OpenSearch connection options: output formats guide.

Run on HuggingFace Jobs

Run evaluations on HF cloud hardware without a local GPU (Pro / Team / Enterprise):

hf jobs run \
  --flavor cpu-basic \
  -s HF_TOKEN=hf_your_token \
  -s OPENAI_API_KEY=your_key \
  python:3.12 \
  bash -c "pip install smoltrace ddgs && smoltrace-eval --model openai/gpt-4 --provider litellm --enable-otel"

GPU flavors (t4-small through a100-large), the Python API, and scheduled evaluations are covered in the HF Jobs guide.

Dataset Management

Each run creates 3 timestamped datasets, so periodic cleanup keeps your profile tidy:

smoltrace-cleanup --older-than 30d            # dry-run preview (default)
smoltrace-cleanup --keep-recent 5 --no-dry-run  # actually delete

Cleanup is dry-run by default, never touches your leaderboard, and permanently protects the benchmark/tasks datasets. See the dataset management guide.

Leaderboard

Every run appends aggregate stats (success rate, avg steps, tokens, duration, cost, CO2, GPU utilization) to your personal leaderboard, with links back to the run's results, traces, and metrics datasets. Community rankings live at huggingface/smolagents-leaderboard — contribute your runs! More in the leaderboard guide.

Documentation

• Quickstart — first evaluation in minutes
• Configuration — environment variables and options
• Running evaluations — providers, difficulty filters, parallelism
• Agent tools — all 20+ tools with security notes
• GPU metrics & cost — what's collected and how it's aggregated
• Changelog
• llms.txt — machine-readable docs index for LLM agents

Community

• Discord — chat with the community (shared with genai-otel-instrument)
• TraceVerse Community — an open org on the Hub for evaluation & observability datasets, spaces, and real-world MCP apps; join and share your own
• TraceMind-AI collection — 40+ domain-specific task datasets to evaluate against
• GitHub Issues — bug reports and feature requests

Contributing

Contributions welcome! See the contributing guide.

git clone https://github.com/Mandark-droid/SMOLTRACE.git && cd SMOLTRACE
pip install -e .[dev]
pytest

License

Apache-2.0. See LICENSE.

⭐ Star this repo to support Smolagents! Questions? Open an issue.