Stoneburner

Local-first LLM evaluation: token cost, quality, and security suites. The same commands cover a laptop Ollama box and a cloud API.

Install stoneburner-atomics. The CLI and the import stay atomics. (atomics on PyPI is a different package; stoneburner is too similar to an existing stone-burner.)

This is a desk tool, not a research harness and not an unsupervised agent. It records cost, quality, and security-suite results in SQLite. A finished-looking percentage on a partial run is the failure mode it is built to avoid: incomplete coverage prints n/a (scored/total scored) and JSON nulls the headline.

atomics doctor ends with one Next: command when the check is healthy. Typical first-run output (Ollama on localhost, no cloud key):

$ atomics doctor
Python 3.13.11 OK
Platform: Darwin (arm64)
Database path: data/atomics.db
SQLite database OK (readable / creatable)
ANTHROPIC_API_KEY not set (optional; needed for Claude)
OPENAI_API_KEY not set (optional; needed for OpenAI)
inference.env: not found (optional; $INFERENCE_ENV or /etc/agentic/inference.env)
Ollama endpoint: http://localhost:11434
Ollama reachable — 3 model(s): qwen2.5:7b, gemma3:4b, llama3.2:3b

Next: atomics provider-test --provider ollama --no-thinking
      Ollama is reachable.

$ atomics toolcall --provider ollama --channel tools --runs 3 --no-thinking

Summary
  tool-capable: yes
  outcomes: safe call=6  no call=14
  channel divergence (resisted in prose, complied with tools): not measured (no qualifying fixtures)
  response divergence (dangerous call, refusing text): not measured (no qualifying fixtures)
  cost: $0.0000

A tools-only first run is valid. Channel divergence needs a second model as judge. Thinking models that spend the token budget on hidden reasoning are recorded as thinking_budget, not as a mystery generation failure.

New here? QUICKSTART — copy-pasteable commands grouped by goal.

Contributing? ARCHITECTURE — layer map, primitives, how to add an eval suite.

Table of Contents

• Install
• Providers
• Burn Tiers
• Key Commands
• Security Suites
• Load Testing
• Thinking Mode
• Configuration
• Secrets Management
• Architecture
• Running Tests
• Further Reading

Install

Ollama on http://localhost:11434 is the one-box path. No cloud key required.

uv tool install stoneburner-atomics
atomics doctor
atomics provider-test --provider ollama --no-thinking
atomics toolcall --provider ollama --channel tools --runs 3 --no-thinking

--no-thinking keeps reasoning models from spending the whole token budget on hidden chain-of-thought. A tools-only toolcall run is a valid first run; channel divergence needs a second model as judge.

API, MCP, or RAG extras:

uv tool install 'stoneburner-atomics[api,mcp]'
uv add 'stoneburner-atomics[rag]'          # from another project

From a clone, uv sync --all-extras. Bare uv sync drops the API, MCP, RAG, and test extras.

Cloud providers work the same way once a key is set:

export ANTHROPIC_API_KEY=sk-ant-...
atomics provider-test
atomics run -n 5
atomics report

atomics run --provider openai -n 5
atomics run --provider bedrock --region us-east-1 -n 5

Providers

Compare providers after running benchmarks: atomics compare — see COMPARING for model classes, metrics fidelity, and judge accuracy.

Optional extras: from PyPI, uv add 'stoneburner-atomics[rag]' (or [bedrock], [openai], [api], [mcp]). From a clone, uv sync --all-extras. RAG is atomics rag-index / rag-retrieval / rag --index. atomics server --with-dashboard serves the read-only UI at /dashboard. atomics mcp proxies a running API server and inherits that server's authentication and spend ceilings. See MCP_SERVER.

Burn Tiers

uv run atomics run --tier ez -n 3 -i 5
uv run atomics tiers                   # show all tier profiles

Key Commands

Full reference: CLI_REFERENCE

Security Suites

Six eval suites for LLM security assessment:

Plus probe (live infrastructure analysis) and sweep (multi-model ranked comparison).

Full documentation: SECURITY_SUITES · Leaderboards: adversarial · red/blue

Load Testing

Full documentation: LOAD_TESTING

Thinking Mode

Auto-detects reasoning-capable models (Claude extended thinking, OpenAI o-series, Ollama qwen3) and tracks thinking tokens separately.

uv run atomics run --provider ollama -m qwen3:14b -n 5   # auto-detected
uv run atomics run --provider claude --thinking -n 5      # explicit
uv run atomics run --provider openai -m o3 --no-thinking  # forced off for A/B

Full documentation: THINKING

Configuration

Set via environment variables (prefix ATOMICS_) or .env file:

Database defaults: macOS: data/atomics.db · Linux: ~/.local/share/atomics/atomics.db (XDG)

CLI flags (--tier, --budget, --interval) override these at runtime.

Secrets Management

Layered resolution: environment variable → .env file → OS keychain (macOS Keychain / Linux secret-service).

atomics secrets set ANTHROPIC_API_KEY   # store securely (hidden input)
atomics secrets list                    # verify
atomics secrets delete ANTHROPIC_API_KEY

Architecture

stoneburner/
├── atomics/              # Core Python package
│   ├── api/              # HTTP API server (FastAPI) — runs, evals, reports, jobs, dashboard
│   ├── commands/         # Click command modules (auth, admin, benchmark, eval, security, load, api, worker, distributed)
│   ├── distributed/      # Coordinator + worker for split and fleet runs
│   │   ├── coordinator.py
│   │   ├── models.py
│   │   ├── worker_client.py
│   │   ├── worker_runner.py
│   │   ├── routes.py
│   │   ├── rollup.py     # Per-worker aggregation of fleet results
│   │   └── auth.py
│   ├── core/             # Loop engine, task runner, rate/budget guard
│   ├── eval/             # Evaluation framework (eval, adversarial, redblue)
│   ├── probe/            # Live ecosystem probe suite
│   ├── archreview/       # Security-architecture repo benchmark
│   ├── providers/        # LLM adapters (Claude, Bedrock, OpenAI, Ollama, vLLM, brain-gateway)
│   ├── storage/          # SQLite metrics persistence (schema v20)
│   ├── scheduler/        # Cron/systemd/launchd generation and installation
│   └── cli.py            # Thin Click root — registers commands from commands/
├── profiles/             # Custom target profiles (local/ gitignored)
├── qa/                   # QA fixture suites (local/ gitignored)
├── tests/                # 2500+ tests
└── docs/                 # Detailed documentation

See ARCHITECTURE for the full layer map and contributor guide.

Running Tests

The api extra is required to run the suite: the API and distributed test modules import FastAPI at module scope, so without it pytest errors during collection instead of skipping. The mcp extra is not required for collection — those tests skip if the SDK is missing — but CI installs it so the MCP surface is actually tested. Sync it locally too.

uv sync --all-extras
uv run pytest -q
uv run pytest -q --cov=atomics --cov-report=term-missing --cov-fail-under=85

The suite drives FastAPI's TestClient, an in-process shim, so it proves the logic but not that the pieces work as separate processes. For that:

uv run python scripts/smoke_fleet.py

This starts a real coordinator and real worker processes, runs a two-host fleet job against a stubbed OpenAI-compatible endpoint, then kills a worker mid-run to confirm the job resolves to partial instead of waiting on a dead host. It needs no credentials and no model, and touches no real database.

Further Reading

Links are absolute so they work on PyPI as well as GitHub.

License

MIT — see LICENSE.