Stoneburner

Local-first LLM evaluation: token cost, quality, and security suites. The same commands cover a laptop Ollama box and a cloud API.

Install stoneburner-atomics. The CLI and the import stay atomics. (atomics on PyPI is a different package; stoneburner is too similar to an existing stone-burner.)

This is a desk tool, not a research harness and not an unsupervised agent. It records cost, quality, and security-suite results in SQLite. A finished-looking percentage on a partial run is the failure mode it is built to avoid: incomplete coverage prints n/a (scored/total scored) and JSON nulls the headline.

atomics doctor ends with one Next: command when the check is healthy. Typical first-run output (Ollama on localhost, no cloud key):

$ atomics doctor
Python 3.13.11 OK
Platform: Darwin (arm64)
Database path: data/atomics.db
SQLite database OK (readable / creatable)
ANTHROPIC_API_KEY not set (optional; needed for Claude)
OPENAI_API_KEY not set (optional; needed for OpenAI)
inference.env: not found (optional; $INFERENCE_ENV or /etc/agentic/inference.env)
Ollama endpoint: http://localhost:11434
Ollama reachable — 3 model(s): qwen2.5:7b, gemma3:4b, llama3.2:3b

Next: atomics provider-test --provider ollama --no-thinking
      Ollama is reachable.

$ atomics toolcall --provider ollama --channel tools --runs 3 --no-thinking

Summary
  tool-capable: yes
  outcomes: safe call=6  no call=14
  channel divergence (resisted in prose, complied with tools): not measured (no qualifying fixtures)
  response divergence (dangerous call, refusing text): not measured (no qualifying fixtures)
  cost: $0.0000

A tools-only first run is valid. Channel divergence needs a second model as judge. Thinking models that spend the token budget on hidden reasoning are recorded as thinking_budget, not as a mystery generation failure.

Install

Ollama on http://localhost:11434 is the one-box path. No cloud key required.

uv tool install stoneburner-atomics
atomics doctor
atomics provider-test --provider ollama --no-thinking
atomics toolcall --provider ollama --channel tools --runs 3 --no-thinking

--no-thinking keeps reasoning models from spending the whole token budget on hidden chain-of-thought.

uv tool install 'stoneburner-atomics[api,mcp]'
uv add 'stoneburner-atomics[rag]'          # from another project

From a clone, uv sync --all-extras. Bare uv sync drops the API, MCP, RAG, and test extras. atomics server, atomics mcp, and atomics repl need those extras and a running API server.

Cloud providers take the same --provider / --effort flags once a key is set. Providers: Claude, Bedrock, OpenAI, Ollama, vLLM, llama.cpp, Groq, Gemini, Together, brain-gateway.

Docs

Recipes live in QUICKSTART. Flags live in CLI_REFERENCE. Links are absolute so they work on PyPI as well as GitHub.

License

MIT — see LICENSE.