Stoneburner

Atomics — Agentic token usage benchmarking platform

A continuous, cron-schedulable benchmarking harness that runs realistic everyday tasks against LLM providers to measure token consumption, cost, throughput, and performance trends over time. Supports tiered usage profiles, multiple providers (including local Ollama), and a full security evaluation suite.

New here? Start with QUICKSTART.md — copy-pasteable commands grouped by goal.

Contributing? Read ARCHITECTURE.md — layer map, primitives, how to add an eval suite.

Table of Contents

• Quick Start
• Providers
• Burn Tiers
• Key Commands
• Security Suites
• Load Testing
• Thinking Mode
• Configuration
• Secrets Management
• Architecture
• Running Tests
• Further Reading

Quick Start

# from a clone
uv sync

# or from PyPI, once a release has been uploaded
uv add stoneburner-atomics
uv tool install stoneburner-atomics

export ANTHROPIC_API_KEY=sk-ant-...

uv run atomics provider-test          # verify connection
uv run atomics run -n 5               # run 5 benchmark tasks
uv run atomics report                 # view results

# other providers
uv run atomics run --provider openai -n 5
uv run atomics run --provider bedrock --region us-east-1 -n 5
uv run atomics run --provider ollama -n 5

Providers

Compare providers after running benchmarks: uv run atomics compare — see docs/COMPARING.md for model classes, metrics fidelity, and judge accuracy.

Optional extras: Real RAG retrieval (atomics rag-index, atomics rag-retrieval, atomics rag --index) requires uv sync --extra rag to install sqlite-vec and sentence-transformers. Bedrock and OpenAI providers need --extra bedrock and --extra openai respectively.

The API server mode requires uv sync --extra api to install FastAPI and uvicorn. Add --with-dashboard to serve a read-only web UI at /dashboard. Serving atomics to LLM agents over MCP (atomics mcp) requires uv sync --extra mcp; it proxies a running API server and inherits that server's authentication and spend ceilings. See docs/MCP_SERVER.md.

Burn Tiers

uv run atomics run --tier ez -n 3 -i 5
uv run atomics tiers                   # show all tier profiles

Key Commands

Full reference: docs/CLI_REFERENCE.md

Security Suites

Six eval suites for LLM security assessment:

Plus probe (live infrastructure analysis) and sweep (multi-model ranked comparison).

Full documentation: docs/SECURITY_SUITES.md · Leaderboards: adversarial · red/blue

Load Testing

Full documentation: docs/LOAD_TESTING.md

Thinking Mode

Auto-detects reasoning-capable models (Claude extended thinking, OpenAI o-series, Ollama qwen3) and tracks thinking tokens separately.

uv run atomics run --provider ollama -m qwen3:14b -n 5   # auto-detected
uv run atomics run --provider claude --thinking -n 5      # explicit
uv run atomics run --provider openai -m o3 --no-thinking  # forced off for A/B

Full documentation: docs/THINKING.md

Configuration

Set via environment variables (prefix ATOMICS_) or .env file:

Database defaults: macOS: data/atomics.db · Linux: ~/.local/share/atomics/atomics.db (XDG)

CLI flags (--tier, --budget, --interval) override these at runtime.

Secrets Management

Layered resolution: environment variable → .env file → OS keychain (macOS Keychain / Linux secret-service).

atomics secrets set ANTHROPIC_API_KEY   # store securely (hidden input)
atomics secrets list                    # verify
atomics secrets delete ANTHROPIC_API_KEY

Architecture

stoneburner/
├── atomics/              # Core Python package
│   ├── api/              # HTTP API server (FastAPI) — runs, evals, reports, jobs, dashboard
│   ├── commands/         # Click command modules (auth, admin, benchmark, eval, security, load, api, worker, distributed)
│   ├── distributed/      # Coordinator + worker for split and fleet runs
│   │   ├── coordinator.py
│   │   ├── models.py
│   │   ├── worker_client.py
│   │   ├── worker_runner.py
│   │   ├── routes.py
│   │   ├── rollup.py     # Per-worker aggregation of fleet results
│   │   └── auth.py
│   ├── core/             # Loop engine, task runner, rate/budget guard
│   ├── eval/             # Evaluation framework (eval, adversarial, redblue)
│   ├── probe/            # Live ecosystem probe suite
│   ├── archreview/       # Security-architecture repo benchmark
│   ├── providers/        # LLM adapters (Claude, Bedrock, OpenAI, Ollama, vLLM, brain-gateway)
│   ├── storage/          # SQLite metrics persistence (schema v20)
│   ├── scheduler/        # Cron/systemd/launchd generation and installation
│   └── cli.py            # Thin Click root — registers commands from commands/
├── profiles/             # Custom target profiles (local/ gitignored)
├── qa/                   # QA fixture suites (local/ gitignored)
├── tests/                # 2289 tests at 89% coverage
└── docs/                 # Detailed documentation

See ARCHITECTURE.md for the full layer map and contributor guide.

Running Tests

The api extra is required to run the suite: the API and distributed test modules import FastAPI at module scope, so without it pytest errors during collection instead of skipping. The mcp extra is not required for collection — those tests skip if the SDK is missing — but CI installs it so the MCP surface is actually tested. Sync it locally too.

uv sync --extra dev --extra api --extra mcp
uv run pytest -q
uv run pytest -q --cov=atomics --cov-report=term-missing --cov-fail-under=85

The suite drives FastAPI's TestClient, an in-process shim, so it proves the logic but not that the pieces work as separate processes. For that:

uv run python scripts/smoke_fleet.py

This starts a real coordinator and real worker processes, runs a two-host fleet job against a stubbed OpenAI-compatible endpoint, then kills a worker mid-run to confirm the job resolves to partial instead of waiting on a dead host. It needs no credentials and no model, and touches no real database.

Further Reading

License

MIT — see LICENSE.