Tiny Agents. Total Visibility.
Smol Agents. Smart Metrics.
Documentation | Quickstart | PyPI | Issues
smoltrace
SMOLTRACE is a benchmarking and evaluation framework for Smolagents, Hugging Face's lightweight agent library. It runs ToolCallingAgent and CodeAgent against curated or custom task datasets, auto-instruments everything with OpenTelemetry, and publishes results, traces, metrics, and a leaderboard — with zero configuration.
Get Started in 30 Seconds
pip install smoltrace
export HF_TOKEN=hf_your_token
export OPENAI_API_KEY=sk-your_key # or MISTRAL_API_KEY, ANTHROPIC_API_KEY, ...
smoltrace-eval --model openai/gpt-4.1-nano --provider litellm --agent-type both --enable-otel
That's it. One command evaluates both agent types and automatically creates 4 datasets on the Hugging Face Hub:
{username}/smoltrace-results-{timestamp}— per-test-case outcomes{username}/smoltrace-traces-{timestamp}— OpenTelemetry spans{username}/smoltrace-metrics-{timestamp}— GPU / CO2 / cost time-series{username}/smoltrace-leaderboard— aggregate stats, updated every run
No repo names to configure — everything is derived from your HF_TOKEN.
Key Features
- Zero configuration — only
HF_TOKENrequired; dataset names are auto-generated from your username - Multi-provider — LiteLLM (OpenAI, Anthropic, Mistral, Groq, ...), HF Inference API, local Transformers models on GPU, and Ollama
- OTEL native — auto-instrumented via genai-otel-instrument: spans, token counts, cost tracking, CO2 emissions
- GPU metrics — utilization, memory, temperature, and power tracked by default for local models, flattened for dashboards
- 20+ agent tools — web search, webpage reading, file system, text processing (
grep/sed/sort), and process/system tools (ps/curl/ping), with working-directory sandboxing - 3 benchmark datasets — quick tasks (13), comprehensive benchmark (132: GAIA/Math/SimpleQA), and an ops/SRE benchmark (24)
- Flexible output — push to HF Hub, save local JSON, or export to OpenSearch with typed mappings
- Parallel execution —
--parallel-workers 8for 10-50x faster API-model evaluations - Dataset lifecycle —
smoltrace-copy-datasetsto bootstrap,smoltrace-cleanupto safely prune old runs - HF Jobs ready — run on Hugging Face cloud hardware (CPU to A100) with pay-as-you-go billing
Installation
pip install smoltrace # core
pip install smoltrace[gpu] # + GPU metrics for local models
pip install smoltrace[opensearch] # + OpenSearch export
Requires Python 3.11+. For development installs and full requirements, see the installation guide.
Usage
CLI
# API model via LiteLLM (default provider)
smoltrace-eval --model openai/gpt-4 --provider litellm --agent-type tool
# Local GPU model via Transformers (GPU metrics on by default)
smoltrace-eval --model meta-llama/Llama-3.1-8B --provider transformers --agent-type both
# Local model via Ollama
smoltrace-eval --model qwen2.5-coder:3b --provider ollama --agent-type tool --enable-otel
# HF Inference API
smoltrace-eval --model meta-llama/Llama-3.1-70B-Instruct --provider inference --agent-type both
# Enable agent tools, filter by difficulty, run in parallel
smoltrace-eval \
--model openai/gpt-4.1-nano \
--provider litellm \
--enable-tools visit_webpage read_file grep \
--working-directory ./workspace \
--difficulty easy \
--parallel-workers 8 \
--enable-otel
Hardware: every run records the machine it ran on in the leaderboard row as hardware (a label such as NVIDIA A10G 22 GB × 1 or CPU 2 vCPU · 16 GB), hardware_detail (CPU model, logical cores, memory, operating system, each GPU) and hardware_scope (model when the model ran on that machine, runner when the model was a hosted service and the machine only drove the run). --hardware-label TEXT replaces the label only, for example with a cloud flavour name; the detail and the scope are still detected. Detection never fails a run.
Generation settings (sent to the model on every call; unset = the provider's default): --temperature, --top-p, --top-k, --max-new-tokens, --reasoning-effort none|minimal|low|medium|high, --enable-thinking true|false. A setting the provider cannot apply is printed as [GENERATION] not applied for <provider>: ... and recorded in the leaderboard row's generation_settings; it is never dropped silently. --enable-thinking is the chat-template switch of open-weight models (OpenAI-compatible endpoints, Ollama, transformers); OpenAI, Anthropic and Gemini natives use --reasoning-effort.
Web search: --enable-tools google_search --search-provider duckduckgo|serper|brave|tavily (SERPER_API_KEY, BRAVE_API_KEY, TAVILY_API_KEY; duckduckgo needs none). A provider without its key stops the run instead of running without search.
Other useful flags: --dataset-name (custom task dataset), --model-args key=value (variables handed to the agent, not to the model), --prompt-yml (custom prompts: a smolagents prompt-template file, e.g. prompt_template.tool.yaml, merged over the defaults; a lone plain system_prompt is prepended to the default; a missing or unparseable file stops the run rather than falling back to the default prompts), --mcp-server-url (MCP tools), --private (private datasets), --output-format hub|json|opensearch.
Full flag reference: CLI documentation.
Python API
import os
from smoltrace.core import run_evaluation
all_results, trace_data, metric_data, dataset_used, run_id = run_evaluation(
model="openai/gpt-4",
provider="litellm",
agent_type="both",
enable_otel=True,
hf_token=os.getenv("HF_TOKEN"),
)
Lower-level building blocks (generate_dataset_names, push_results_to_hf, compute_leaderboard_row, update_leaderboard, ...) are documented in the Python API reference.
Benchmark Datasets
| Dataset | Size | Purpose |
|---|---|---|
kshitijthakkar/smoltrace-tasks (default) |
13 tasks | Quick validation and development |
kshitijthakkar/smoltrace-benchmark-v1 |
132 tasks | Comprehensive evaluation (GAIA, Math, SimpleQA) |
kshitijthakkar/smoltrace-ops-benchmark |
24 tasks | APM / AIOps / SRE / DevOps scenarios |
Copy them to your own account with smoltrace-copy-datasets, or bring your own tasks as a HF dataset or local JSON. Schemas, the ops-benchmark sample-data setup, and custom task creation are covered in the datasets guide.
Community task datasets: the TraceMind-AI collection bundles 40+ ready-to-run task datasets spanning real-world domains — travel, e-commerce, healthcare, finance, DevOps/SRE, AIOps, cybersecurity, and more — for evaluating agents on domain-specific workloads. Point --dataset-name at any of them, e.g. --dataset-name MCP-1st-Birthday/smoltrace-finance-tasks.
Output Formats
--output-format hub # default: 4 datasets on the HF Hub
--output-format json # 5 local JSON files (results, traces, metrics, leaderboard row, metadata)
--output-format opensearch # 4 OpenSearch indexes with typed mappings and bulk indexing
Details and OpenSearch connection options: output formats guide.
Run on HuggingFace Jobs
Run evaluations on HF cloud hardware without a local GPU (Pro / Team / Enterprise):
hf jobs run \
--flavor cpu-basic \
-s HF_TOKEN=hf_your_token \
-s OPENAI_API_KEY=your_key \
python:3.12 \
bash -c "pip install smoltrace ddgs && smoltrace-eval --model openai/gpt-4 --provider litellm --enable-otel"
GPU flavors (t4-small through a100-large), the Python API, and scheduled evaluations are covered in the HF Jobs guide.
Dataset Management
Each run creates 3 timestamped datasets, so periodic cleanup keeps your profile tidy:
smoltrace-cleanup --older-than 30d # dry-run preview (default)
smoltrace-cleanup --keep-recent 5 --no-dry-run # actually delete
Cleanup is dry-run by default, never touches your leaderboard, and permanently protects the benchmark/tasks datasets. See the dataset management guide.
Leaderboard
Every run appends aggregate stats (success rate, pass@1, avg steps, tokens, duration, cost, CO2, GPU utilization) to your personal leaderboard, with links back to the run's results, traces, and metrics datasets. SMOLTRACE computes pass@1 from the first recorded completion for each logical (agent_type, test_id) task and publishes the rule and denominator alongside the score. Community rankings live at huggingface/smolagents-leaderboard — contribute your runs! More in the leaderboard guide.
Documentation
- Quickstart — first evaluation in minutes
- Configuration — environment variables and options
- Running evaluations — providers, difficulty filters, parallelism
- Agent tools — all 20+ tools with security notes
- GPU metrics & cost — what's collected and how it's aggregated
- Changelog
- llms.txt — machine-readable docs index for LLM agents
Community
- Discord — chat with the community (shared with genai-otel-instrument)
- TraceVerse Community — an open org on the Hub for evaluation & observability datasets, spaces, and real-world MCP apps; join and share your own
- TraceMind-AI collection — 40+ domain-specific task datasets to evaluate against
- GitHub Issues — bug reports and feature requests
Contributing
Contributions welcome! See the contributing guide.
git clone https://github.com/Mandark-droid/SMOLTRACE.git && cd SMOLTRACE
pip install -e .[dev]
pytest
License
Apache-2.0. See LICENSE.
⭐ Star this repo to support Smolagents! Questions? Open an issue.
Metadata
Release files for smoltrace 0.2.12
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| smoltrace-0.2.12.tar.gz | 185.1 kB | Without provenance | Details for smoltrace-0.2.12.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| smoltrace-0.2.12-py3-none-any.whl | Python 3 | none | any | Details for smoltrace-0.2.12-py3-none-any.whl |
Total release size: 291.3 kB
Release files / smoltrace-0.2.12.tar.gz
| Download URL | smoltrace-0.2.12.tar.gz |
|---|---|
| Size | 185.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8655275db4e77ce75f907ca9a9c957782a1d2c0e194a8437de5749091ef31833
|
|
BLAKE2b-256 checksum How to use checksums |
d668648cbc051f693d560bdfa1a19c5cdd301192368b1fb7582da0df92550a52
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.17
|
Release files / smoltrace-0.2.12-py3-none-any.whl
| Download URL | smoltrace-0.2.12-py3-none-any.whl |
|---|---|
| Size | 106.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d513669a0f50816b1fe39fc2bcde881d9ef14b21f7c8aa76ee47516e2b7395c2
|
|
BLAKE2b-256 checksum How to use checksums |
76c9bb24be49b40cf2d04fd769c3706db811f6f3b94f53232bda2158f8b89e44
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.17
|
Security for smoltrace 0.2.12
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.