Astraea Atomic

Developed by Astraea. Atomic tests AI models on procedurally generated construction problems with exact solution checking.

The terminal evaluator runs a selected Claude model against a fixed reference configuration and reports an Atomic Constraint rating (experimental) with a 95% confidence interval. It also includes an offline SDK for trying individual challenges.

Run a model evaluation

You need Python 3.10 or newer, uv, Docker installed and running, and an Anthropic API key with credit. The controlled Python evaluator requires macOS or Linux and a local Docker Unix socket. Native Windows evaluation is not supported in this release. Local release validation uses macOS ARM64 and Python 3.12; the Python version declaration is not a claim that every platform/version combination has been tested.

Start the evaluator:

uvx --from astraea-atomic==0.1.0 atomic evaluate

The terminal shows Atomic branding and guides you through selecting a model, entering your API key without echoing it, choosing the number of puzzles and setting a spending limit. It prepares the isolated Python runtime, shows progress, and prints your result. No Python file or Astraea account is needed.

An existing ANTHROPIC_API_KEY environment variable is reused. Run evidence is saved locally under ./atomic-results/ by default; --output PATH chooses a new directory.

Recheck a saved run without making API calls:

atomic report PATH_TO_RUN_DIRECTORY

The report checks the saved puzzle generation, trial records, accounting and score. This checks local consistency, not independent authenticity. Generation seeds stay in a private evaluator file and are never given to either model.

Both the selected model and the reference make paid API calls using your key. The default run uses 32 fresh puzzle pairs: 64 model trials, with Sonnet 5.5 at high effort anchored at 1,000 points. The default local accounting limit is $31, covering the conservative reservation of approximately $30.07 for all 32 pairs. This reservation is not an expected bill. Historical Sonnet runs suggest approximately $4.48 in returned usage for 32 pairs, but different models, responses, provider prices and errors can change the cost. The local limit is not an account-wide provider billing cap.

The evaluator downloads an official Python 3.12 container image when needed and records the local immutable image ID used for the run. Model-written code executes in restricted Docker containers; it has no network or host-folder access. API requests are made by the evaluator outside those containers.

Alternatively, install into a virtual environment on macOS/Linux:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install astraea-atomic==0.1.0
atomic evaluate

See the complete terminal walkthrough.

Understand your result

The experimental rating compares candidate-only successes with reference-only successes on the same fresh 384-variable Constraint Lock puzzles, using the versioned controlled Python protocol. The reference is assigned 1,000 by convention; it is not a measure of absolute intelligence.

When both counts are nonzero, the score is 1000 + 400 × log10(candidate-only successes / reference-only successes). Its 95% interval comes from an exact conditional binomial interval for the direction of the discordant pairs. Matching outcomes do not identify the rating difference. The numerical scale is uncapped, but the fixed 384-variable release is not proven to remain informative for arbitrarily strong future models.

For a complete run, the evaluator reports the observed pass rates and one of:

• Estimate and 95% interval: both models have at least one unique success.
• An upper or lower bound: unique successes occurred in only one direction, so a finite point estimate is unavailable.
• Unidentified: all paired outcomes matched; this does not establish equal ability.

An interrupted or otherwise incomplete run is identified as incomplete and does not receive a rating from its completed prefix. A run with few puzzles can produce a wide interval or only a bound. More puzzles can improve precision, but do not guarantee a finite estimate.

The uncertainty calculation is conditional on the comparison's statistical assumptions. It does not include every possible source of uncertainty, establish that all model families behave alike, or make the rating a general intelligence score. Results include skill at writing code, using tools and staying within the declared token and execution budgets. Self-run results are not an independently certified leaderboard entry.

Earlier research required a 95% interval no wider than 200 points. The completed study failed that target. This release exposes experimental estimates with their uncertainty rather than withholding every imprecise estimate; it does not retroactively mark that study as passing. Its results and amendments remain in the research record.

Try the free offline demo

No API key, Docker or model calls are needed for the tutorials:

uvx --from astraea-atomic==0.1.0 atomic demo

The demo verifies two supplied example answers. It produces an unrated practice scorecard, not an AI evaluation or experimental rating.

For local development:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install .
atomic demo
atomic evaluate

Use the practice SDK

from astraea_atomic import Atomic
from astraea_atomic.examples import switchyard_solution

atomic = Atomic()
env = atomic.make("switchyard-tutorial", render_mode="terminal")
env.submit(switchyard_solution())
print(atomic.get_scorecard())

The supplied solution demonstrates the interface. Practice scorecards retain status: "unrated" and atomic_score: null; calling the SDK does not enforce an evaluation budget or certify a model score.

Generate and inspect a reproducible practice problem:

atomic make constraint-research --seed 42 --level 1 --output challenge.json
atomic verify constraint-research solution.json --seed 42 --level 1

For Constraint Lock, submit {"assignment": [...]} with one integer 0 or 1 per variable. Any satisfying assignment is accepted. Verification scans the clauses without searching for a solution. Levels select structural sizes, not calibrated difficulty ratings.

The SDK also includes Switchyard and Counterexample Forge tutorials and research generators. They are not combined into the experimental Constraint rating. Larger problems and an uncapped numerical formula do not establish that the benchmark remains informative indefinitely.

Research and development

The earlier internal README is preserved in the research history in the source workspace. Historical studies, provider records, private answers, credentials and local budget ledgers are excluded from the public package.

python -m pip install -e .
python -m unittest discover -s tests -v

Development tests belong to the full development workspace; they are not included in the public source distribution. The installed Python package has no third-party Python runtime dependencies. Model evaluation separately requires Docker, network access to Anthropic and paid API access.