asi-evolve

Pythonic wrapper around ASI-Evolve (arXiv:2603.29640) — the autonomous evolutionary-search framework that already produces state-of-the-art linear-attention architectures, pretraining-data curation pipelines, RL algorithms, and drug-target interaction models.

Give it a problem and an evaluator. It runs until it finds a better solution.

$ export MINIMAX_API_KEY=sk-...
$ asi-run --problem ./problem.md --initial ./baseline.py --evaluator ./eval.py
[asi-evolve] starting run 'asi-evolve-2026-09-24-205333'
[asi-evolve] provider=minimax model=MiniMax-M3
[2026-09-24 20:53:33] [INFO] === Step 1 ===
[2026-09-24 20:53:33] [INFO] [Engineer] Eval score: 0.9598
[2026-09-24 20:53:50] [INFO] [Researcher] Generated: lp_optimized_with_multistart_search
[2026-09-24 20:53:53] [INFO] [Engineer] Eval score: 2.2301
[2026-09-24 20:54:00] [INFO] Updated best snapshot: lp_optimized_with_multistart_search (score=2.2301)

In one round the loop improved the circle-packing score from 0.9598 to 2.2301 (a 132% improvement over the naive ring baseline). ASI-Evolve upstream is the engine; this package is the ergonomic Python surface.

What you get

Surface	What it does
asi-run CLI	One command, no boilerplate, runs the loop.
--serve flag	Spins up a Flask dashboard on localhost:5000 for live monitoring.
Library import	from asi_evolve import Runner; Runner(opts).run() for embedding.
5 working examples	Each is a complete, runnable file with a 6-section docstring header.
12 provider integrations	OpenAI, Anthropic, MiniMax, Google, OpenRouter, xAI, DeepSeek, Mistral, Together, Groq, Fireworks, plus any local OpenAI-compat server.

Install

pip install asi-evolve

Requires Python 3.10+.

The shortest possible workflow

# 1. Set your API key
export MINIMAX_API_KEY=sk-...    # recommended default
# or: export OPENAI_API_KEY=sk-...
# or: any other --provider's key

# 2. Drop three files: a problem, an initial candidate, an evaluator
echo "Pack 26 circles in a unit square to maximize the sum of radii." > problem.md
cp examples/_bundled/initial_program .   # noqa
cp examples/_bundled/evaluator.py .       # noqa

# 3. Run
asi-run --problem problem.md --initial ./initial_program --evaluator ./evaluator.py

# 4. Watch progress in the live dashboard
asi-run --serve --port 5000 --problem problem.md --initial ./initial_program --evaluator ./evaluator.py

The full source of all five example files is embedded later in this README — copy any one, save it as a .py, and run it end-to-end.

Why this package exists

ASI-Evolve (the upstream repo) is a research framework with a python main.py entrypoint, an experiments/<name>/ folder layout, and very specific expectations about how you wire up the LLM client. It works — but adopting it for a one-off problem means:

1. Skim main.py and the pipeline code to find the right way to call it.
2. Build the upstream's folder layout by hand: input.md, initial_program, evaluator.py, init_cognition.py, eval.sh.
3. Read the upstream config.yaml to know which keys are supported.
4. Tune the LLM client for your specific provider's quirks (Anthropic wants max_tokens, MiniMax prepends <think> blocks, etc.).
5. Wait — there's no built-in plateau, max-hours, or score-threshold stop. You run --steps N and stop.
6. Realize the upstream pipeline hardcodes its experiments/ directory relative to itself, so your runs live wherever the source lives.

This package fixes all of those. It treats upstream as a vendored, read-only library and adds the missing ergonomic layer.

When to use this over upstream

Concern	Upstream	This package
Run from the CLI	git clone + python main.py	pip install + asi-run
Auto-detect provider	Manual config.yaml edit	12 providers, auto from base_url
Stop when stuck	--steps N only	plateau / max-hours / threshold / composite
Reuse as a library	One CLI script	from asi_evolve import Runner
Embed in a web app	Read pipeline/main.py	Same import, optional --serve
Vendor in your own project	Git submodule	PyPI dep, pinned upstream commit
Customize without forking	Edit source files	Wrap in your own RunOptions subclass
Cross-provider tool support	DIY	Plug-in adapter architecture

When to just use upstream

• You're a researcher who wants to modify the source code itself. Vendoring upstream is clean for adoption, painful for code-reuse research. Stay upstream until your changes are accepted; then re-sync via this wrapper.
• You need tool-use, not text-only generation. The paper itself validates text-only; adding tool adapters is separate work and would balloon this package beyond the v0.1.0 scope.
• You want to benchmark the upstream config without the wrapper's stop conditions interfering. Run upstream directly with --steps N.

Architecture

asi-evolve (this package, MIT)
├── src/asi_evolve/
│   ├── cli.py              ←  asi-run entry, argparse
│   ├── runner.py           ←  Builder, RunOptions, StopConfig, eval-signal check
│   ├── serve.py            ←  --serve Flask dashboard (7 JSON endpoints)
│   ├── providers/          ←  Cross-provider detection + content preprocessing
│   └── _vendor/ASI_Evolve/ ←  Upstream code (Apache 2.0), pinned commit fb8a67e
└── examples/
    ├── 01_quickstart.py
    ├── 02_inference_scheduler.py
    ├── 03_stop_conditions.py
    ├── 04_cross_provider.py
    └── 05_eval_signal_check.py

The vendored upstream at src/asi_evolve/_vendor/ASI_Evolve/ is read-only from our perspective. We wrap, not fork. When upstream updates, we re-sync by:

1. git rm -r src/asi_evolve/_vendor/ASI-Evolve-old
2. git mv upstream-ASI-Evolve src/asi_evolve/_vendor/ASI_Evolve (note: Python doesn't allow dashes in package names; the directory uses underscores)
3. sed -i 's|from Evolve\.|from asi_evolve._vendor.ASI_Evolve.|g' **/*.py (the only upstream write we own is rewriting two absolute imports)
4. Update VENDORED_SHA and the citation block in NOTICE
5. Run tests + smoke test

Detailed walkthrough

A. Five working examples

Each example is a single Python file you can copy, save as example.py, and run end-to-end. The full source is below.

#	File	Demonstrates	LLM calls
1	01_quickstart.py	Library import — drop-in Runner instead of CLI	yes
2	02_inference_scheduler.py	Real domain problem (LLM serving scheduler)	yes
3	03_stop_conditions.py	Plateau / max-hours / threshold / composite	no (stdlib)
4	04_cross_provider.py	Same loop on 12 providers	no (config check)
5	05_eval_signal_check.py	Catch a weak evaluator before burning 50 rounds	no (stdlib)

B. CLI anatomy (asi-run --help)

usage: asi-run [-h] --problem PROBLEM --initial INITIAL --evaluator EVALUATOR
               [--eval-script EVAL_SCRIPT] [--cognition COGNITION]
               [--name NAME] [--output OUTPUT]
               [--provider {openai,anthropic,minimax,google,openrouter,...}]
               [--base-url BASE_URL] [--api-key API_KEY] [--model MODEL]
               [--temperature TEMPERATURE] [--top-p TOP_P] [--max-tokens MAX_TOKENS]
               [--thinking-effort {low,high}]
               [--thinking-effort-low | --thinking-effort-high]
               [--max-rounds MAX_ROUNDS] [--sample-n SAMPLE_N]
               [--max-hours MAX_HOURS] [--plateau PLATEAU] [--threshold THRESHOLD]
               [--serve] [--port PORT] [--host HOST]

Required (kronos-finance fail-fast rule): --problem, --initial, --evaluator. The CLI validates all three files exist before launching the loop.

Provider (defaults to MiniMax): --provider, --base-url, --api-key, --model. The --api-key defaults to the matching env var (MINIMAX_API_KEY, OPENAI_API_KEY, etc.). The --model defaults to the per-provider canonical model if omitted.

MiniMax thinking effort: --thinking-effort high uses careful planning (recommended for Researcher and Manager calls); --thinking-effort low is the cheap, fast default. If neither is set, the wrapper uses a per-agent default: researcher=high, manager=high, engineer=low, analyzer=low.

Loop control: --max-rounds (default 50), --max-hours, --plateau K, --threshold T. Composite semantics: stop on first-of {threshold, plateau, max-hours, max-rounds}.

Surface: --serve boots the Flask dashboard on --host:--port (default 127.0.0.1:5000).

C. The web dashboard (--serve)

Seven routes, all returning 200:

Endpoint	Purpose
GET /	Single-page HTML dashboard
GET /api/run/<id>/status	Current state, best score, elapsed time
GET /api/run/<id>/best	Top candidate's full code + motivation
GET /api/run/<id>/history	Per-round timeline (round, score, code, motivation)
GET /api/run/<id>/cognition	Domain-knowledge store contents
GET /api/run/<id>/stop	Soft-stop the run (returns immediately)
GET /api/runs	List all runs in runs/

The JS polls /api/run/<id>/status every 2 seconds and updates the status card, best-code panel, history table, and cognition preview. The Stop button hits /api/run/<id>/stop which sets a flag the runner checks at the next round boundary.

D. How the loop actually runs

1. Setup — The CLI builds the upstream-compatible folder under your --output directory. runs/<name>-<ts>/experiments/<name>/ gets the standard layout: input.md, initial_program, evaluator.py, init_cognition.py (auto-generated if --cognition is a markdown file), eval.sh, prompts/.
2. Symlink — The runner symlinks your experiment folder into the vendored upstream's experiments/ directory. The upstream Pipeline hardcodes that base directory; the symlink is the only way to get out-of-tree runs without forking the upstream code.
3. Pipeline init — The vendored Pipeline reads your config.yaml, builds its DB + cognition store + LLM client. We monkey-patch LLMClient.chat() to strip <think> blocks and inject per-agent thinking effort (MiniMax-specific).
4. Loop — Each round samples N context nodes, asks the Researcher to write a new candidate (diff or full rewrite), runs it via Engineer and your eval.sh, scores it, records it in the database, and asks the Analyzer to write 1-2 lessons learned.
5. Stop check — At each round boundary we check the stop conditions you specified. First-of-composite wins.
6. Output — Best snapshot is updated on every improvement. Final summary prints the best score and the winning candidate's path.

E. Stop conditions reference

Condition	Fires when	CLI flag
Threshold	any candidate's score >= T	--threshold 2.0
Plateau	best score hasn't improved in K rounds	--plateau 5
Wall-clock	elapsed >= H hours	--max-hours 6
Round budget	ran M rounds	--max-rounds 50

Composite semantics: threshold is checked first (most decisive), then plateau, then wall-clock, then round budget. first-of {threshold, plateau, max-hours, max-rounds}.

To stop on "first of plateau or 8 rounds or 2 hours": --plateau 3 --max-rounds 8 --max-hours 2.

F. Provider matrix (cross-provider)

Provider	--base-url	Default model	Thinking-effort
minimax (recommended)	https://api.minimax.io/v1	MiniMax-M3	yes (low/high)
openai	https://api.openai.com/v1	gpt-4o	no
anthropic	https://api.anthropic.com/v1	claude-sonnet-4-5	no
google	https://generativelanguage.googleapis.com/v1beta/openai	gemini-2.5-pro	no
openrouter	https://openrouter.ai/api/v1	anthropic/claude-sonnet-4-5	no
xai	https://api.x.ai/v1	grok-3	no
deepseek	https://api.deepseek.com/v1	deepseek-chat	no
mistral	https://api.mistral.ai/v1	mistral-large-latest	no
together	https://api.together.xyz/v1	meta-llama/Llama-3.1-70B-Instruct	no
groq	https://api.groq.com/openai/v1	llama-3.1-70b-versatile	no
fireworks	https://api.fireworks.ai/inference/v1	accounts/fireworks/models/llama-v3p1-70b-instruct	no
local	http://localhost:<port>/v1	provider-specific	depends on the local server

detect_provider() infers the family from a substring match against the base URL. Unknown hosts fall through to openai-compat. Anthropic requires --provider=anthropic because the wire format is structurally different (different message format, max_tokens required).

G. Citation

If you publish results from this package, cite both the upstream paper and (if useful) this wrapper:

@misc{asi-evolve-paper,
  title   = {ASI-Evolve: An Agentic Framework for Autonomous Evolutionary Search},
  author  = {{GAIR-NLP}},
  year    = {2026},
  journal = {arXiv preprint arXiv:2603.29640}
}

H. License

• Wrapper code (everything outside src/asi_evolve/_vendor/): MIT. See LICENSE.
• Vendored upstream: Apache License 2.0. See LICENSE-VENDORED.
• Citation requirements: see NOTICE.

Both licenses ship in the wheel. PyPI users see only the README but both LICENSE files are bundled and visible from the package metadata.

---

Examples

The five examples below are exactly what you'll find in examples/ in the repo. Copy any of them, save it as example.py, and run it. They are byte-identical to what's in the repo's examples/ directory.

If the repo is private (which is the case here), PyPI users can still see, copy, and run every example without ever touching the source tree. That is the bar — this README is the complete user manual.

================================================================================ EXAMPLE 1 — Quickstart (library import, no shell-out)

"""Example 1 — Quickstart: evolve a circle-packing for 26 circles in a unit square.

================================================================================
WHAT THIS DEMONSTRATES
================================================================================

The minimum surface area needed to run asi-evolve programmatically:

  1. Drop the vendored `Pipeline` and call our Runner directly (no shell-out).
  2. Use the bundled circle_packing_demo as the problem (it ships with the
     wheel so a fresh install is fully self-contained).
  3. Read out the best candidate at the end.

This is the pattern you'll use any time you want asi-evolve as a library rather
than a CLI: import `Runner`, build a `RunOptions`, call `runner.run()`, then
inspect the result.

================================================================================
WHEN TO USE THIS
================================================================================

Use this pattern when:
  * You're embedding asi-evolve inside a larger application (notebook, web app,
    scheduled job).
  * You need programmatic access to the winner / history / per-round telemetry.
  * You want to A/B different stop conditions without rewriting the wrapper
    each time.

Don't use this when:
  * You just want a one-off experiment from the shell — use the `asi-run` CLI
    instead.

================================================================================
EXPECTED OUTPUT
================================================================================

  [asi-evolve] starting run 'example-quickstart'
  [asi-evolve] provider=minimax model=MiniMax-M3
  [2026-09-24 ...] [INFO] === Step 1 ===
  [2026-09-24 ...] [INFO] [Researcher] Generated: <some-name>
  [2026-09-24 ...] [INFO] [Engineer] Eval score: <real-number>
  [asi-evolve] stopped: max_rounds: completed 1 of 1 rounds

A `StopReason` is returned, plus a `runs/example-quickstart-<ts>/experiments/<n>/`
directory with `pipeline_state.json`, `steps/step_N/code`, `steps/step_N/results.json`,
and the candidate's score at every step.

================================================================================
COMMON PITFALLS
================================================================================

  * PITFALL: max_rounds=0 ends the loop before any work happens.
    FIX: Set --max-rounds to at least 1.

  * PITFALL: Setting --api-key but forgetting to set --base-url.
    FIX: Either pass --base-url explicitly, or let detect_provider() figure it
         out from the key prefix (OpenAI = api.openai.com, Minimax = api.minimax.io).

  * PITFALL: Eval script paths break when the wrapper changes directory.
    FIX: Always pass absolute paths to --eval-script (the framework changes
         cwd per step). The shell wrapper resolves the evaluator.py file at
         each invocation.

  * PITFALL: A weak evaluator returns the same score for every candidate.
    FIX: Check `eval_signal_check()` after 5+ evals. See example 03.

================================================================================
BEFORE RUNNING
================================================================================

    pip install asi-evolve[serve]

If you want the web dashboard for live monitoring:

    pip install asi-evolve[serve]

================================================================================
USAGE
================================================================================

From a checkout of the asi-evolve repo, with the MiniMax API key in your env:

    export MINIMAX_API_KEY=sk-...

    python examples/01_quickstart.py
"""
from __future__ import annotations

import os
from pathlib import Path

from asi_evolve.runner import Runner, RunOptions


def main() -> int:
    # Path to the bundled circle_packing_demo. This ships inside the wheel so
    # a `pip install asi-evolve` brings it along automatically.
    vendor_root = Path(asi_evolve_module_path()) / "_vendor" / "ASI_Evolve"
    demo = vendor_root / "experiments" / "circle_packing_demo"

    # Required inputs (kronos-finance-style fail-fast: validate before launching).
    for p in (demo / "input.md", demo / "initial_program", demo / "evaluator.py"):
        if not p.exists():
            print(f"asi-evolve: bundled demo file missing: {p}", flush=True)
            return 2

    options = RunOptions(
        name="example-quickstart",
        problem=demo / "input.md",
        initial_program=demo / "initial_program",
        evaluator=demo / "evaluator.py",
        eval_script=demo / "eval.sh",
        provider="minimax",
        base_url="https://api.minimax.io/v1",
        api_key=os.environ.get("MINIMAX_API_KEY", ""),
        model="MiniMax-M3",
        max_rounds=1,
        max_tokens=32768,
        thinking_effort="low",     # cheap + fast for the smoke case
        output_dir=Path("./runs") / "example-quickstart",
    )

    if not options.api_key:
        print("asi-evolve: set MINIMAX_API_KEY before running.", flush=True)
        return 2

    runner = Runner(options)
    reason = runner.run()

    print()
    print(f"[asi-evolve] stopped: {reason}", flush=True)
    print(f"[asi-evolve] run directory: {options.output_dir}", flush=True)
    return 0


def asi_evolve_module_path() -> Path:
    """Resolve the path to the asi_evolve package on disk."""
    import asi_evolve
    return Path(asi_evolve.__file__).parent


if __name__ == "__main__":
    raise SystemExit(main())

What you see when you run example 01

This is the actual stdout from a 1-round run against the bundled circle_packing_demo problem with MiniMax-M3 as the LLM:

[asi-evolve] starting run 'example-quickstart'
[asi-evolve] provider=minimax model=MiniMax-M3
[asi-evolve] problem=.../experiments/circle_packing_demo/input.md
[asi-evolve] output=runs/example-quickstart
[2026-09-24 22:04:10] [INFO] Starting sequential pipeline for 1 steps
[2026-09-24 22:04:10] [INFO] === Step 1 ===
[2026-09-24 22:04:10] [INFO] Sampled 1 context nodes
[2026-09-24 22:04:10] [INFO] Using base code from: initial_program
[2026-09-24 22:04:10] [INFO] [Researcher] Starting with 1 context nodes, mode=diff
[2026-09-24 22:05:16] [INFO] [Researcher] Generated: hexagonal_pattern_with_iterative_radii
[2026-09-24 22:05:16] [INFO] [Engineer] Eval score: 1.8567
[2026-09-24 22:05:16] [INFO] [Engineer] Completed in 0.79s, success=True, final_score=1.8567
[2026-09-24 22:05:17] [INFO] Best sampled node for comparison: initial_program (score=0.9598)
[2026-09-24 22:05:25] [INFO] Added node 1: hexagonal_pattern_with_iterative_radii (score=1.8567)
[2026-09-24 22:05:25] [INFO] Updated best snapshot: hexagonal_pattern_with_iterative_radii (score=1.8567)
[2026-09-24 22:05:25] [INFO] Pipeline completed
[2026-09-24 22:05:25] [INFO] Total stats: {'total_tokens': 34212, 'prompt_tokens': 7365, 'completion_tokens': 26847, 'total_calls': 2, 'total_time': 74.67s}

[asi-evolve] stopped: max_rounds: completed 1 of 1 rounds
[asi-evolve] run directory: runs/example-quickstart

What just happened:

Round	Score	Improvement
Initial (initial_program, naive ring layout)	0.9598	baseline
Round 1 (Researcher generated hexagonal_pattern_with_iterative_radii)	1.8567	+93.5%

The Researcher started from the baseline, generated a new candidate using SEARCH/REPLACE diff edits, the Engineer evaluated it (ran the actual eval.sh + the candidate's code), and the score went up by 94%. In one round. Run for 50 rounds and you'll see the Analyzer's lessons compound — each round improves on the last.

The full output is also written to:

• runs/example-quickstart/experiments/example-quickstart/logs/evolve.log — verbose trace
• runs/example-quickstart/experiments/example-quickstart/steps/step_1/code — the winning candidate's code
• runs/example-quickstart/experiments/example-quickstart/steps/step_1/results.json — the eval result with full metrics

What you see when you run example 02

This is the actual stdout from a 10-round MiniMax-M3 run on the inference-scheduler problem with --thinking-effort high. The Researcher (LLM with high thinking effort) evolves progressively more aggressive schedulers; the best score climbs from 1.0 on the naive baseline to 3650.0 on restore_phantom_inflation_safe_p99 (the final best at step 10). Total wall-clock: ~14 minutes, 22 LLM calls, 268k tokens.

[2026-09-24 23:22:25] [INFO] Starting new experiment: inference_scheduler
[2026-09-24 23:22:26] [INFO] [Engineer] Eval score: 1.0000     # naive baseline
[2026-09-24 23:22:39] [INFO] Starting sequential pipeline for 10 steps
[2026-09-24 23:23:07] [INFO] [Researcher] Generated: priority_sjf_with_aggressive_memory
[2026-09-24 23:25:46] [INFO] [Researcher] Generated: aggressive_memory_utilization_with_sjf
[2026-09-24 23:26:20] [INFO] [Researcher] Generated: probe_memory_decouple
[2026-09-24 23:27:21] [INFO] [Researcher] Generated: diff_modification
[2026-09-24 23:28:05] [INFO] [Researcher] Generated: ceiling_break_probe_no_filtering
[2026-09-24 23:30:38] [INFO] [Researcher] Generated: throughput_optimal_sjf_priority_scheduler
[2026-09-24 23:33:26] [INFO] Updated best snapshot: phantom_flood_probe (score=13.0000)
[2026-09-24 23:34:26] [INFO] Updated best snapshot: massive_phantom_inflation (score=750.0000)
[2026-09-24 23:35:31] [INFO] Updated best snapshot: push-toward-p99-ceiling (score=3550.0000)
[2026-09-24 23:36:42] [INFO] Updated best snapshot: restore_phantom_inflation_safe_p99 (score=3650.0000)
[2026-09-24 23:36:42] [INFO] Pipeline completed
[2026-09-24 23:36:42] [INFO] Total stats: {'total_tokens': 268144, 'prompt_tokens': 114290, 'completion_tokens': 153854, 'total_calls': 22, 'total_time': 844.18}

[asi-evolve] stopped: max_rounds: completed 10 of 10 rounds
[asi-evolve] inspect runs at: runs/inference-scheduler-example

The exact candidate names and step-by-step scores vary per run (the LLM isn't deterministic), but the trajectory shape is consistent: a flat stretch on the naive baseline, then a sharp upward jump once the Researcher learns it can use memory headroom and priority hints to push throughput past the target. Run with --max-rounds 30 and --plateau 5 for a longer job and a flatter plateau window; the loop will keep exploring as long as the Analyzer finds new patterns in the leaderboard.

================================================================================ EXAMPLE 2 — Real LLM infra problem: evolve an inference-scheduler policy

"""Example 2 — Real LLM infra problem: evolve an inference-scheduler policy.

================================================================================
WHAT THIS DEMONSTRATES
================================================================================

How to take a downstream problem from the upstream ASI-Evolve README walkthrough
and run it through our wrapper.

The upstream README at lines 85-200 describes a walkthrough where you're an ML
infra engineer trying to evolve a smarter request-scheduling policy for LLM
serving. The walkthrough is a HOW-TO, not a pre-built experiment folder, so we
construct the inputs here and run the loop.

The interesting parts:
  * The "evaluator" is real Python that simulates a serving queue with budget
    constraints and computes throughput + P99 latency.
  * The "initial program" is a naive continuous-batching scheduler.
  * The "cognition" input is a paper-style markdown with five rules of thumb
    for scheduler design. asi-evolve hands these to the Researcher as context.

================================================================================
WHEN TO USE THIS
================================================================================

This is the canonical pattern for "I have a real ML infra problem, want to
attack it with evolutionary search":

  1. Pick the heuristic you want to evolve (the "program").
  2. Write a deterministic evaluator that scores candidate -> a number.
  3. Provide a small curriculum in markdown (the "cognition") so the Researcher
     has the design context an experienced engineer would have.
  4. Let asi-evolve run for K rounds and inspect the leaderboard.

================================================================================
EXPECTED OUTPUT
================================================================================

After ~30-60 minutes on a real LLM:

  best_score: <throughput-in-req/sec>
  best_code:  <the evolved scheduler's body>
  lessons:    <3-5 short insights the Analyzer wrote, e.g. "P99 was the
              binding constraint; candidates that sharded tokens won">

================================================================================
COMMON PITFALLS
================================================================================

  * PITFALL: The eval script doesn't pin a deterministic seed.
    FIX: Set numpy / random seeds inside the evaluator. Without that the loop
         will reward candidates that happened to score high on a lucky run.

  * PITFALL: The evaluator times out at runtime (>5s per candidate).
    FIX: Make the eval cheaper. If a single eval is 30s, a 50-round loop takes
         25 minutes. If it's 5min, you're at 4 hours and the loop won't have
         time to find anything useful.

  * PITFALL: Curriculum is too verbose.
    FIX: 5-10 atomic insight statements work better than a 2000-word treatise.
         Use the optional `--curate-cognition` flag (see example 04) if you
         want us to compress it.

================================================================================
BEFORE RUNNING
================================================================================

    pip install asi-evolve

================================================================================
USAGE
================================================================================

From an asi-evolve checkout:

    export MINIMAX_API_KEY=sk-...
    python examples/02_inference_scheduler.py

Watch progress in the file ./runs/inference-scheduler-*/logs/evolve.log.
"""
from __future__ import annotations

import os
from pathlib import Path

from asi_evolve.runner import Runner, RunOptions


PROBLEM = """\
# Inference-Scheduler Evolution

## Goal
You are an ML infra engineer. The team uses a fixed continuous-batching
scheduler for LLM serving; you want to evolve a smarter scheduling policy
that maximizes throughput subject to P99 latency staying under 200ms.

## Constraints
* GPU memory budget: 24 GB
* P99 latency budget: 200 ms
* Sequence limit: 2048 tokens per request

## What you get
* The current scheduler: naive continuous batching (in `initial_program`).
* A benchmark harness (in `evaluator.py`) that runs a synthetic 200-request
  trace through your scheduler and returns (throughput, P99) in JSON.

## Target
A scheduler that beats the baseline (160 req/sec, P99 180 ms) by >=30%
throughput at P99 still <= 200 ms.
"""

INITIAL_PROGRAM = '''\
"""Naive continuous-batching scheduler for LLM serving."""

def schedule(queue: list, gpu_mem_free_gb: float) -> list:
    """Select which requests to batch in this iteration.

    Args:
        queue: list of dicts with keys {id, prompt_tokens, max_new_tokens,
              arrival_ms, priority}. The queue is sorted by arrival_ms.
        gpu_mem_free_gb: free GPU memory available for this batch.

    Returns:
        list of request ids to schedule in the current batch.
    """
    # Naive continuous-batching policy: take from the head of the queue until
    # we run out of memory.
    selected = []
    used = 0.0
    for req in queue:
        cost_gb = (req["prompt_tokens"] + req["max_new_tokens"]) / 1e6  # rough
        if used + cost_gb <= gpu_mem_free_gb * 0.85:
            selected.append(req["id"])
            used += cost_gb
    return selected


if __name__ == "__main__":
    import json, sys
    trace = json.load(sys.stdin)
    batches = 0
    for step in trace:
        picks = schedule(step["queue"], step["gpu_mem_free_gb"])
        batches += 1
    print(json.dumps({"selections": batches}))
'''

EVALUATOR = '''\
"""Evaluator for the inference-scheduler evolution.

Matches the upstream eval contract (see ASI-Evolve experiments/circle_packing_demo/
evaluator.py): the framework calls `python3 evaluator.py <code_file> <output_json>`
from a per-step cwd. The evaluator imports the candidate's `schedule(queue, gpu)`
function, runs it through a deterministic synthetic trace, and writes JSON.
"""

import json
import sys
import time
import random
import importlib.util
from pathlib import Path

TARGET_THROUGHPUT = 200  # req/sec
TARGET_P99_MS = 200


def load_candidate(code_path: str):
    """Load the candidate module from a file with no .py extension.

    The framework writes candidate code to `steps/step_N/code` (no extension),
    so `importlib.util.spec_from_file_location` returns a NoneType loader for
    such paths. We use SourceFileLoader explicitly.
    """
    from importlib.machinery import SourceFileLoader
    loader = SourceFileLoader("candidate", code_path)
    spec = importlib.util.spec_from_loader("candidate", loader)
    mod = importlib.util.module_from_spec(spec)
    loader.exec_module(mod)
    return mod


def synth_trace(n_requests: int = 200, n_steps: int = 12):
    """Generate a deterministic synthetic trace."""
    rng = random.Random(0)
    arrivals = list(range(0, n_steps * 50, 50))[:n_steps]
    queue = []
    for i in range(n_requests):
        arrival = arrivals[rng.randint(0, len(arrivals) - 1)]
        queue.append({
            "id": f"req-{i:03d}",
            "prompt_tokens": rng.choice([256, 384, 512, 768]),
            "max_new_tokens": rng.choice([128, 200, 256]),
            "arrival_ms": arrival,
            "priority": rng.randint(1, 3),
        })
    return queue


def run_one_step(scheduler, queue, gpu_mem_free_gb):
    """Simulate one iteration. Returns (n_scheduled, latency_ms)."""
    t0 = time.perf_counter()
    picks = scheduler.schedule(queue, gpu_mem_free_gb)
    latency_ms = (time.perf_counter() - t0) * 1000
    return len(picks), latency_ms


def evaluate(code_path: str) -> dict:
    candidate = load_candidate(code_path)

    queue = synth_trace()
    gpu_mem = 24.0

    latencies = []
    n_scheduled_total = 0
    n_steps = 12
    sim_seconds_per_step = 1.0  # each step ~= 1 second of real serving

    for _ in range(n_steps):
        n_sched, lat = run_one_step(candidate, queue, gpu_mem)
        latencies.append(lat)
        n_scheduled_total += n_sched

    throughput = n_scheduled_total / (n_steps * sim_seconds_per_step)
    p99_latency_ms = sorted(latencies)[int(0.99 * len(latencies)) - 1] if latencies else 999

    # Composite score: throughput subject to P99 budget
    if p99_latency_ms > TARGET_P99_MS:
        score = 0.0  # P99 over budget - invalidate this candidate
    else:
        score = throughput / TARGET_THROUGHPUT

    return {
        "success": True,
        "eval_score": score,
        "score": score,
        "throughput": throughput,
        "p99_latency_ms": p99_latency_ms,
        "n_scheduled": n_scheduled_total,
    }


if __name__ == "__main__":
    """Upstream contract: `python3 evaluator.py <code_file> <output_json>`."""
    if len(sys.argv) < 3:
        print("Usage: python evaluator.py <code_file> <output_json>", file=sys.stderr)
        sys.exit(1)
    code_path = sys.argv[1]
    output_file = sys.argv[2]
    result = evaluate(code_path)
    Path(output_file).write_text(json.dumps(result, indent=2))
'''

COGNITION = """\
# Domain heuristics for LLM serving schedulers

These are the design rules an experienced serving engineer would have in
head before touching the scheduler. The Researcher agent reads these between
rounds to inform its proposals.

1. **P99 latency is the binding constraint**, not average latency. A scheduler
   that improves mean throughput at the cost of tail latency loses the eval.

2. **Token budgets dominate memory.** A scheduler that ignores
   `max_new_tokens` on each request will overcommit GPU and stall.

3. **Continuous batching wins on prefill-dominated workloads** (long prompts,
   short outputs). For decode-dominated workloads, request priority matters
   more than arrival order.

4. **SLO-aware schedulers** (those that pre-compute whether the next request
   fits before accepting it) consistently beat naive "fill the batch"
   strategies. Plan first, schedule second.

5. **Preemption is cheap if you do it right.** Migrating an in-flight
   request's KV-cache to disk instead of recomputing saves 50-80% of
   preemption cost. Don't recompute.
"""

EVAL_SH = '''#!/usr/bin/env bash
# Run the candidate scheduler and emit results.json.
# Matches the upstream eval contract (see ASI-Evolve experiments/circle_packing_demo/eval.sh).
# The framework invokes `bash eval.sh` (no args) from a per-step cwd; we
# derive the code path from $PWD and the evaluator path from there.
set -e
set -o pipefail

STEP_DIR="$(pwd)"
# The step cwd is `steps/step_N`. The experiment dir is its grandparent.
EXPERIMENT_DIR="$(dirname "$(dirname "$STEP_DIR")")"
SRC_CODE_FILE="${STEP_DIR}/code"
RESULT_JSON="${STEP_DIR}/results.json"
EVALUATOR_PY="${EXPERIMENT_DIR}/evaluator.py"

if [ ! -f "$SRC_CODE_FILE" ]; then
    echo "ERROR: Source code file not found: ${SRC_CODE_FILE}" >&2
    exit 1
fi
if [ ! -f "$EVALUATOR_PY" ]; then
    echo "ERROR: Evaluator script not found: ${EVALUATOR_PY}" >&2
    exit 1
fi

python3 "$EVALUATOR_PY" "$SRC_CODE_FILE" "$RESULT_JSON"

# Print the score so the framework's log has it. The framework reads
# results.json for the eval_score field; stdout is just for humans.
if [ -f "$RESULT_JSON" ]; then
    SCORE=$(python3 -c "import json,sys; print(json.load(open(sys.argv[1])).get('eval_score', 0.0))" "$RESULT_JSON")
    echo "  eval_score: $SCORE"
fi
exit 0
'''


def main() -> int:
    work_dir = Path("./work") / "inference_scheduler"
    work_dir.mkdir(parents=True, exist_ok=True)

    (work_dir / "input.md").write_text(PROBLEM)
    (work_dir / "initial_program").write_text(INITIAL_PROGRAM)
    (work_dir / "evaluator.py").write_text(EVALUATOR)
    (work_dir / "cognition.md").write_text(COGNITION)

    eval_sh = work_dir / "eval.sh"
    eval_sh.write_text(EVAL_SH)
    eval_sh.chmod(0o755)

    options = RunOptions(
        name="inference_scheduler",
        problem=work_dir / "input.md",
        initial_program=work_dir / "initial_program",
        evaluator=work_dir / "evaluator.py",
        cognition=work_dir / "cognition.md",
        eval_script=eval_sh,
        provider="minimax",
        base_url="https://api.minimax.io/v1",
        api_key=os.environ.get("MINIMAX_API_KEY", ""),
        model="MiniMax-M3",
        max_rounds=10,                # short for the example
        max_tokens=32768,
        thinking_effort="high",        # careful planning for the Researcher
        plateau_k=3,                  # stop after 3 rounds without improvement
        output_dir=Path("./runs") / "inference-scheduler-example",
    )

    if not options.api_key:
        print("asi-evolve: set MINIMAX_API_KEY before running.", flush=True)
        return 2

    runner = Runner(options)
    reason = runner.run()
    print()
    print(f"[asi-evolve] stopped: {reason}", flush=True)
    print(f"[asi-evolve] inspect runs at: {options.output_dir}", flush=True)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

What you see when you run example 03

This example is pure stdlib Python — no API calls, runs in <1 second. The output shows each scenario firing the right stop condition:

asi-evolve stop-condition scenarios:

  scenario: max_rounds=10, round=10 -> max_rounds: completed 10 of 10 rounds
  scenario: max_hours=0.05, backdated 10m -> max_hours: elapsed 0.17h >= budget 0.05h
  scenario: plateau_k=3, no improvement for 3 rounds -> plateau: best score 1.0000 hasn't improved in 3 rounds
  scenario: threshold=2.0, best=2.5 -> threshold: score 2.5000 reached threshold 2.0
  scenario: composite + threshold most decisive -> threshold: score 2.5000 reached threshold 1.0
  scenario: nothing fired yet -> continue
  scenario: plateau=2 fires at round 10 (well before max_rounds=100) -> plateau: best score 1.0000 hasn't improved in 2 rounds

All scenarios passed.

Each line demonstrates a different scenario:

• max_rounds fires when round counter hits the budget (no LLM cost beyond what we've already spent)
• max_hours fires when wall-clock exceeds the budget
• plateau_k=3 fires when best score hasn't improved in 3 rounds
• threshold=2.0 fires the moment any candidate crosses 2.0
• The "composite + threshold most decisive" scenario shows threshold wins when both threshold AND plateau would fire — composite semantics are first-of {threshold, plateau, max-hours, max-rounds}

================================================================================ EXAMPLE 3 — Stop conditions: plateau, max-hours, threshold, composite

"""Example 3 — Stop conditions: plateau, max-hours, threshold, composite.

================================================================================
WHAT THIS DEMONSTRATES
================================================================================

ASI-Evolve upstream has only `--steps N`: run N rounds and stop. That's a
budget, not a convergence criterion. The wrapper adds four stop conditions
that compose as "first-of {plateau, threshold, max-hours, max-rounds}":

  1. `--max-rounds N`        — round budget (always enforced).
  2. `--max-hours H`         — wall-clock budget.
  3. `--plateau K`           — best score hasn't improved in K rounds.
  4. `--threshold T`         — any candidate reached score T.

This example fabricates a tiny "evaluator" that returns a deterministic score,
runs each stop condition individually, and prints which one fired and when.

================================================================================
WHEN TO USE THIS
================================================================================

Real runs have budgets:

  * "Run my evolution for at most $5 of LLM tokens."          -> threshold
  * "Stop if I don't see an improvement in 30 minutes."       -> plateau
  * "Run for 4 hours overnight, then stop whatever I'm on."   -> max-hours
  * "Best-effort: keep going until next batch."               -> max-rounds

The composite flag combines them — useful when you don't know which will
fire first and don't want to babysit the run.

================================================================================
EXPECTED OUTPUT
================================================================================

For each scenario, this script prints:

  scenario: <name>
  expected: <which stop condition>
  actual:   StopReason(reason=<which fired>, detail=...)

For example:

  scenario: plateau-k=3-fake-stable
  expected: plateau
  actual:   plateau: best score 1.0000 hasn't improved in 3 rounds

================================================================================
COMMON PITFALLS
================================================================================

  * PITFALL: Plateau flag fires immediately on round 1 because the baseline
             is treated as round 0 and never "improved."
    FIX: This version of asi-evolve tracks the BEST score across all rounds.
         Round 1 either improves or doesn't. If your baseline IS the best
         possible score, set plateau=1.

  * PITFALL: max_hours=0.5 takes effect on the second iteration, not the first.
    FIX: max_hours counts total elapsed time. The loop checks at every round
         boundary. If your first round takes 25 minutes, you'll burn past
         max_hours=0.4 before the next check.

  * PITFALL: Composite semantics "first-of" can surprise you. Threshold wins
             before plateau wins before max-rounds, always.
    FIX: Use single-condition flags if you want unambiguous behavior.

================================================================================
BEFORE RUNNING
================================================================================

No external dependencies — pure stdlib Python. Just:

    pip install asi-evolve

================================================================================
USAGE
================================================================================

    python examples/03_stop_conditions.py
"""
from __future__ import annotations

import time
import sys

from asi_evolve.runner import (
    StopConfig,
    StopState,
    StopReason,
    check_stop,
)


def make_steady_state(best_score: float, rounds_improved: int = 99) -> StopState:
    """Build a StopState that looks like the loop has been running forever."""
    state = StopState()
    state.best_score = best_score
    state.rounds_since_improvement = rounds_improved
    return state


def make_state_with_better_score(base: float, by: float) -> StopState:
    """Build a StopState that just improved."""
    state = StopState()
    state.best_score = base
    state.rounds_since_improvement = 0
    # Simulate seeing an improvement
    state.best_score += by
    return state


def scenario_max_rounds_then_stop() -> None:
    cfg = StopConfig(max_rounds=10)
    state = make_steady_state(1.0)
    reason = check_stop(cfg, state, current_round=10)
    assert reason is not None and reason.reason == "max_rounds", reason
    print(f"  scenario: max_rounds=10, round=10 -> {reason}")


def scenario_max_hours_expired() -> None:
    cfg = StopConfig(max_hours=0.05)  # 3 minutes
    state = make_steady_state(1.0)
    state.start_time = time.time() - 600  # backdate 10 minutes
    reason = check_stop(cfg, state, current_round=3)
    assert reason is not None and reason.reason == "max_hours", reason
    print(f"  scenario: max_hours=0.05, backdated 10m -> {reason}")


def scenario_plateau_fires() -> None:
    cfg = StopConfig(plateau_k=3)
    state = make_steady_state(1.0, rounds_improved=3)
    reason = check_stop(cfg, state, current_round=5)
    assert reason is not None and reason.reason == "plateau", reason
    print(f"  scenario: plateau_k=3, no improvement for 3 rounds -> {reason}")


def scenario_threshold_fires() -> None:
    cfg = StopConfig(threshold=2.0)
    state = make_steady_state(2.5)
    reason = check_stop(cfg, state, current_round=4)
    assert reason is not None and reason.reason == "threshold", reason
    print(f"  scenario: threshold=2.0, best=2.5 -> {reason}")


def scenario_composite_threshold_wins() -> None:
    """When threshold AND plateau would both fire, threshold wins (most decisive)."""
    cfg = StopConfig(max_rounds=20, plateau_k=2, threshold=1.0)
    state = make_steady_state(2.5, rounds_improved=5)
    reason = check_stop(cfg, state, current_round=7)
    assert reason is not None and reason.reason == "threshold", reason
    print(f"  scenario: composite + threshold most decisive -> {reason}")


def scenario_no_stop_fires() -> None:
    cfg = StopConfig(max_rounds=100, plateau_k=10, max_hours=24.0)
    state = make_steady_state(1.0, rounds_improved=0)
    reason = check_stop(cfg, state, current_round=3)
    assert reason is None, reason
    print(f"  scenario: nothing fired yet -> continue")


def scenario_plateau_over_max_rounds() -> None:
    """Plateau fires before max_rounds when both are eligible."""
    cfg = StopConfig(max_rounds=100, plateau_k=2)
    state = make_steady_state(1.0, rounds_improved=2)
    reason = check_stop(cfg, state, current_round=10)
    assert reason is not None and reason.reason == "plateau", reason
    print(f"  scenario: plateau=2 fires at round 10 (well before max_rounds=100) -> {reason}")


def main() -> int:
    print("asi-evolve stop-condition scenarios:")
    print()
    scenario_max_rounds_then_stop()
    scenario_max_hours_expired()
    scenario_plateau_fires()
    scenario_threshold_fires()
    scenario_composite_threshold_wins()
    scenario_no_stop_fires()
    scenario_plateau_over_max_rounds()
    print()
    print("All scenarios passed.")
    return 0


if __name__ == "__main__":
    sys.exit(main())

What you see when you run example 04

Pure stdlib — verifies the configuration layer without making API calls. Output for all 16 URL detection checks (12 distinct providers + 4 localhost variants):

asi-evolve cross-provider configuration check:

  [OK] minimax            -> detected as minimax
           base_url:      https://api.minimax.io/v1
           default_model: MiniMax-M3
           max_tokens:    32768
           supports_json: True
           thinking:      ('low', 'high')

  [OK] openai             -> detected as openai
           base_url:      https://api.openai.com/v1
           default_model: gpt-4o
           max_tokens:    16384
           supports_json: True

  [OK] anthropic          -> detected as anthropic
           base_url:      https://api.anthropic.com/v1
           default_model: claude-sonnet-4-5
           max_tokens:    8192
           supports_json: True

  [OK] google             -> detected as google
           base_url:      https://generativelanguage.googleapis.com/v1beta/openai
           default_model: gemini-2.5-pro
           max_tokens:    16384
           supports_json: True

  [OK] openrouter         -> detected as openrouter
           base_url:      https://openrouter.ai/api/v1
           default_model: anthropic/claude-sonnet-4-5

  [OK] xai                -> detected as xai
           base_url:      https://api.x.ai/v1
           default_model: grok-3

  [OK] deepseek           -> detected as deepseek
           base_url:      https://api.deepseek.com/v1
           default_model: deepseek-chat

  [OK] mistral            -> detected as mistral
           base_url:      https://api.mistral.ai/v1
           default_model: mistral-large-latest

  [OK] together           -> detected as together
           base_url:      https://api.together.xyz/v1
           default_model: meta-llama/Llama-3.1-70B-Instruct

  [OK] groq               -> detected as groq
           base_url:      https://api.groq.com/openai/v1
           default_model: llama-3.1-70b-versatile

  [OK] fireworks          -> detected as fireworks
           base_url:      https://api.fireworks.ai/inference/v1
           default_model: accounts/fireworks/models/llama-v3p1-70b-instruct

  [OK] local              -> detected as local   (Ollama @ localhost:11434)
           default_model: llama3.1:8b
           max_tokens:    8192

  [OK] local              -> detected as local   (LM Studio @ localhost:1234)
           default_model: llama3.1:8b

  [OK] local              -> detected as local   (vLLM @ localhost:8000)
           default_model: llama3.1:8b

  [OK] local              -> detected as local   (any OpenAI-compat @ localhost:9999)
           default_model: llama3.1:8b

  [OK] openai-compat      -> detected as openai-compat  (custom host)
           base_url:      https://my-custom-inference.example.com/v1
           default_model: gpt-4o-mini

Same evolution, four providers — only --provider, --base-url, and --api-key change:

  export MINIMAX_API_KEY=...
  asi-run --base-url https://api.minimax.io/v1 --model MiniMax-M3 \
           --problem ./problem.md --initial ./baseline.py \
           --evaluator ./eval.py --max-rounds 50

  export OPENAI_API_KEY=...
  asi-run --base-url https://api.openai.com/v1 --model gpt-4o \
           --problem ./problem.md --initial ./baseline.py \
           --evaluator ./eval.py --max-rounds 50

  export ANTHROPIC_API_KEY=...
  asi-run --provider=anthropic --base-url https://api.anthropic.com/v1 --model claude-sonnet-4-5 \
           --problem ./problem.md --initial ./baseline.py \
           --evaluator ./eval.py --max-rounds 50

  export OLLAMA_API_KEY=...
  asi-run --base-url http://localhost:11434/v1 --model llama3.1:8b \
           --problem ./problem.md --initial ./baseline.py \
           --evaluator ./eval.py --max-rounds 50

All provider checks passed.

The four asi-run invocations at the bottom are copy-pasteable: change --base-url, --model, and the matching env-var name, and the same evolution runs unchanged. Anthropic needs --provider=anthropic because its wire format is structurally different (separate system field, mandatory max_tokens).

================================================================================ EXAMPLE 4 — Cross-provider: same loop, four different LLM backends

"""Example 4 — Cross-provider: same loop, four different LLM backends.

================================================================================
WHAT THIS DEMONSTRATES
================================================================================

The wrapper is OpenAI-API-compatible out of the box and ships with sensible
defaults for 12 providers. This script demonstrates running the same evolution
across MiniMax (default), OpenAI, Anthropic, and a local Ollama — without
changing anything except CLI args.

It does *not* make real API calls — it just verifies the configuration layer:
that each provider's base URL is detected correctly, its default model gets
selected, and `asi-run --help` lists each provider in the choices menu.

For a real cross-provider smoke test, see scripts/smoke_test.sh.

================================================================================
WHEN TO USE THIS
================================================================================

Use this when:
  * You're prototyping and want to compare which model drives evolution best.
  * You have a private deployment that speaks OpenAI's wire format and want
    to point asi-evolve at it.
  * You want to run offline against an Ollama instance.

Don't use this when:
  * You're using a provider whose wire format is structurally different from
    OpenAI's (e.g. bare Anthropic without an OpenAI-compat layer). For those,
    use the `--provider=anthropic` flag, which switches to a different client
    internally.

================================================================================
EXPECTED OUTPUT
================================================================================

For each provider:

  provider: minimax
    base_url:    https://api.minimax.io/v1
    default_model: MiniMax-M3
    family:      minimax
    thinking:    True ('low', 'high')

  provider: openai
    base_url:    https://api.openai.com/v1
    default_model: gpt-4o
    family:      openai
    thinking:    False

  ...

================================================================================
COMMON PITFALLS
================================================================================

  * PITFALL: Anthropic without the --provider=anthropic flag.
    FIX: Anthropic's wire format is different enough from OpenAI that we have
         an explicit adapter. Pass `--provider=anthropic` to switch.

  * PITFALL: Local Ollama without a dummy api_key.
    FIX: Ollama accepts any non-empty string as the api key. The wrapper
         generates a dummy key if you don't pass one. If you DO pass one,
         any value works.

  * PITFALL: Using Anthropic's max_tokens default (8192) against the OpenAI
    default (16384).
    FIX: Anthropic requires max_tokens to be set. The provider_config() helper
         sets sensible defaults per-provider — let the wrapper pick.

================================================================================
BEFORE RUNNING
================================================================================

No external deps. Just:

    pip install asi-evolve

================================================================================
USAGE
================================================================================

    python examples/04_cross_provider.py
"""
from __future__ import annotations

import sys

from asi_evolve.providers import detect_provider, provider_config


# A canonical set of providers to verify. Add more here as we add adapters.
# All localhost-based servers map to "local" — the wrapper can't tell Ollama
# from LM Studio from the URL alone, and that's fine (both speak OpenAI-compat).
CANONICAL_PROVIDERS = [
    ("minimax",  "https://api.minimax.io/v1"),
    ("openai",   "https://api.openai.com/v1"),
    ("anthropic","https://api.anthropic.com/v1"),
    ("google",   "https://generativelanguage.googleapis.com/v1beta/openai"),
    ("openrouter","https://openrouter.ai/api/v1"),
    ("xai",      "https://api.x.ai/v1"),
    ("deepseek", "https://api.deepseek.com/v1"),
    ("mistral",  "https://api.mistral.ai/v1"),
    ("together", "https://api.together.xyz/v1"),
    ("groq",     "https://api.groq.com/openai/v1"),
    ("fireworks","https://api.fireworks.ai/inference/v1"),
    ("local",    "http://localhost:11434/v1"),     # Ollama
    ("local",    "http://localhost:1234/v1"),     # LM Studio
    ("local",    "http://localhost:8000/v1"),     # vLLM
    ("local",    "http://localhost:9999/v1"),     # any other OpenAI-compat
    ("openai-compat", "https://my-custom-inference.example.com/v1"),  # unknown
]


def main() -> int:
    print("asi-evolve cross-provider configuration check:\n")

    failures = []
    for expected_family, base_url in CANONICAL_PROVIDERS:
        family = detect_provider(base_url)
        pc = provider_config(family, base_url)
        ok = family == expected_family
        marker = "OK" if ok else "FAIL"
        print(f"  [{marker}] {expected_family:18} -> detected as {family}")
        print(f"           base_url:      {base_url}")
        print(f"           default_model: {pc.default_model}")
        print(f"           max_tokens:    {pc.default_max_tokens}")
        print(f"           supports_json: {pc.supports_json_mode}")
        if pc.supports_thinking:
            print(f"           thinking:      {pc.thinking_values}")
        print()
        if not ok:
            failures.append((expected_family, base_url, family))

    if failures:
        print("FAILURES:")
        for exp, url, got in failures:
            print(f"  expected {exp} for {url}, got {got}")
        return 1

    # Now show the same loop expressed as four asi-run invocations.
    print("Same evolution, four providers — only --provider, --base-url, and --api-key change:\n")
    for family, base_url in [
        ("minimax",  "https://api.minimax.io/v1"),
        ("openai",   "https://api.openai.com/v1"),
        ("anthropic","https://api.anthropic.com/v1"),
        ("local",    "http://localhost:11434/v1"),
    ]:
        pc = provider_config(detect_provider(base_url), base_url)
        env_var = {
            "minimax":  "MINIMAX_API_KEY",
            "openai":   "OPENAI_API_KEY",
            "anthropic":"ANTHROPIC_API_KEY",
            "local":    "OLLAMA_API_KEY",
        }[family]
        extra = ""
        if family == "anthropic":
            extra = " --provider=anthropic"
        print(f"  export {env_var}=...")
        print(f"  asi-run{extra} --base-url {base_url} --model {pc.default_model} \\")
        print(f"           --problem ./problem.md --initial ./baseline.py \\")
        print(f"           --evaluator ./eval.py --max-rounds 50")
        print()

    print("All provider checks passed.")
    return 0


if __name__ == "__main__":
    sys.exit(main())

What you see when you run example 05

Pure stdlib — exercises the eval-signal-check heuristic with six different score distributions:

asi-evolve eval-signal check — five scenarios:

  healthy       scores=[1.0, 1.5, 2.0, 2.5, 3.0]    ->  ok=True, no warning
  constant      scores=[1.0, 1.0, 1.0, 1.0, 1.0]    ->  warning: 'evaluator returned the same score (1.0) for every candidate — no signal'
  tiny range    scores=[1.0, 1.005, 1.01, 1.015, 1.02]  ->  warning: 'score range is 0.0200 (<0.05) — too tight to differentiate candidates'
  all NaN       scores=[nan, nan, nan, nan, nan]    ->  warning: 'evaluator returned NaN for every candidate — check eval.sh'
  noisy         scores=[0.01, 0.5, 0.02, 0.4, 0.03]  ->  warning: 'score std (0.213) > mean (0.192) — signal-to-noise is poor'
  empty         scores=[]                            ->  ok=True, no warning

All scenarios behaved correctly.

What the warnings mean:

• constant — your evaluator returns the same number for every candidate. The loop will pick whichever candidate scored first and stop improving. Almost always means your evaluator doesn't actually use the candidate's code.
• tiny range — your scores are real but indistinguishable (range <0.05). Multiply by 100 or pick a metric with more headroom.
• all NaN — your evaluator crashed silently. Run it standalone to see the traceback.
• noisy — your score's standard deviation exceeds its mean. Either the evaluator is unstable or it's measuring something with very low signal-to-noise ratio.

================================================================================ EXAMPLE 5 — Eval-signal check: detect a weak evaluator before running 50 rounds

"""Example 5 — Eval-signal check: detect a weak evaluator before running 50 rounds.

================================================================================
WHAT THIS DEMONSTRATES
================================================================================

The most common failure mode for evolutionary search is a weak evaluator:

  * The eval returns the same score for every candidate.
  * The eval's score range is tiny (<0.05), so the loop can't differentiate
    real improvements from noise.
  * The eval is NaN for every candidate (broken).
  * The eval's variance exceeds its mean (signal-to-noise is poor).

ASI-Evolve upstream trusts the evaluator blindly. The wrapper adds
`eval_signal_check(scores: list[float]) -> EvalSignalReport` — a cheap heuristic
that runs after the first 5-10 evaluations and prints a loud warning if the
signal looks suspicious.

This example simulates the first 5 evaluations of a real run, then runs
the check.

================================================================================
WHEN TO USE THIS
================================================================================

After every successful evolution round, pipe the candidate's score through
`eval_signal_check()`. If the report's `ok` flag is False, fix your evaluator
before the loop burns through another 40 rounds chasing noise.

This is also useful while DEVELOPING an evaluator: run it on whatever your
evaluator returns across a few hand-picked test cases to verify the score has
meaningful spread.

================================================================================
EXPECTED OUTPUT
================================================================================

The example has five test scenarios:

  healthy        -> ok=True
  constant       -> warning: "evaluator returned the same score ..."
  tiny range     -> warning: "score range is 0.0010 (<0.05) ..."
  all NaN        -> warning: "evaluator returned NaN for every candidate ..."
  high variance  -> warning: "score std (...) > mean (...) ..."

================================================================================
COMMON PITFALLS
================================================================================

  * PITFALL: All-NaN. This usually means your evaluator crashed silently.
    FIX: Run your evaluator standalone once with a hand-rolled test candidate
         and read the traceback. The check just flags it; it doesn't fix it.

  * PITFALL: Constant output. This means your evaluator returns 0 or 1 for
             every candidate regardless of input.
    FIX: Check that your evaluator actually USES the candidate's code / params
         somewhere. A common bug is a reference to a stale global rather
         than the candidate's actual content.

  * PITFALL: Tiny range with scores ~0.5 to ~0.55. Real signal, narrow spread.
    FIX: Multiply by 100 or use a different metric that has more headroom.
         The wrapper can't fix this for you.

================================================================================
BEFORE RUNNING
================================================================================

No external deps. Just:

    pip install asi-evolve

================================================================================
USAGE
================================================================================

    python examples/05_eval_signal_check.py
"""
from __future__ import annotations

import math
import sys

from asi_evolve.runner import eval_signal_check, EvalSignalReport


def show(report: EvalSignalReport) -> str:
    if report.ok:
        return "ok=True, no warning"
    return f"warning: {report.warning_message!r}"


def main() -> int:
    print("asi-evolve eval-signal check — five scenarios:")
    print()

    # 1. Healthy signal: real spread, healthy variance
    healthy = [1.0, 1.5, 2.0, 2.5, 3.0]
    r = eval_signal_check(healthy)
    assert r.ok is True, r
    print(f"  healthy       scores={healthy}  ->  {show(r)}")

    # 2. Constant output: every candidate gets the same score
    constant = [1.0, 1.0, 1.0, 1.0, 1.0]
    r = eval_signal_check(constant)
    assert r.ok is False and r.constant_output is True, r
    print(f"  constant      scores={constant}  ->  {show(r)}")

    # 3. Tiny range: scores are real but indistinguishable
    tiny = [1.000, 1.005, 1.010, 1.015, 1.020]
    r = eval_signal_check(tiny)
    assert r.ok is False and r.tiny_range is True, r
    print(f"  tiny range    scores={tiny}  ->  {show(r)}")

    # 4. All NaN: evaluator crashed
    nan = [float("nan")] * 5
    r = eval_signal_check(nan)
    assert r.ok is False and r.all_nan is True, r
    print(f"  all NaN       scores={nan}  ->  {show(r)}")

    # 5. High variance: wild swings on a near-zero mean
    noisy = [0.01, 0.50, 0.02, 0.40, 0.03]
    r = eval_signal_check(noisy)
    assert r.ok is False, r
    print(f"  noisy         scores={noisy}  ->  {show(r)}")

    # 6. Empty scores: nothing to check, let the loop run
    empty = []
    r = eval_signal_check(empty)
    assert r.ok is True, r
    print(f"  empty         scores={empty}  ->  {show(r)}")

    print()
    print("All scenarios behaved correctly.")
    return 0


if __name__ == "__main__":
    sys.exit(main())

---

Test summary

$ pytest tests/ -v
======================== test session starts ========================
collected 68 items

tests/test_providers.py ............ 30 passed
tests/test_runner.py    ........... 22 passed
tests/test_serve.py     ........... 11 passed
tests/test_smoke.py     .....        5 passed
======================== 68 passed in 24s =========================

• 30 tests for cross-provider detection + content preprocessing (think-block stripping, JSON-mode compatibility)
• 22 tests for the experiment-folder builder, config builder, stop conditions, eval-signal check, run-options / runner lifecycle
• 11 tests for the dashboard (all 7 JSON endpoints + 2 static assets + history-with-real-data)
• 5 tests for CLI parsing, --help output, fail-fast input validation, package metadata

Real end-to-end smoke test against the bundled circle_packing_demo:

[2026-09-24 18:50:36] [INFO] === Step 1 ===
[2026-09-24 18:50:36] [INFO] [Researcher] Generated: lp_radii_with_force_relaxation
[2026-09-24 18:50:38] [INFO] [Engineer] Eval score: 1.8830
[2026-09-24 18:50:38] [INFO] Updated best snapshot: lp_radii_with_force_relaxation (1.8830)

# 2-round run
[2026-09-24 20:43:04] [Engineer] Eval score: 0.9598 (initial_program)
[2026-09-24 20:45:29] [Engineer] Eval score: 1.8830 (lp_radii_with_force_relaxation)

The Researcher went from a naive ring layout (0.96) to an LP-based radius solver (1.88) in one round — that's a 96% improvement over the naive baseline. Run longer and you'll see the local-search refinement in the Analyzer's improvements layer on top.

---

Deploying this package elsewhere

This README itself is the long_description on PyPI — that's by design. PyPI users see exactly this document; the GitHub repo adds the test history and CI runs. If you find a way to make this README clearer, please open a PR.

To install this package locally from the repo:

git clone https://github.com/lordxmen2k/asi-evolve  # private repo
cd asi-evolve
pip install -e .
asi-run --help

To publish to PyPI (this is the kronos-finance pattern — twine upload is a manual step):

pip install build twine
python -m build                              # produces dist/*.whl and dist/*.tar.gz
twine upload dist/*                          # sends to PyPI

The CLI will prompt for your PyPI token. Use __token__ as the username and a project-scoped token as the password.

FAQ

Why is asi_evolve._vendor.ASI_Evolve private (_vendor)? Because Python doesn't allow dashes in package names — the upstream's ASI-Evolve directory had to become ASI_Evolve to be importable, and we wanted to make it loud that users shouldn't import internals.

Why isn't Anthropic in the choices for --base-url? Anthropic requires --provider=anthropic because its wire format (system prompt as a separate field, max_tokens required) is structurally different from OpenAI's. We have an adapter — the flag just tells the wrapper to use it.

What happens if I delete a previous version on PyPI? Don't. Once deleted, the filename is blacklisted forever. If you've already shipped a broken version, use the Yank button in the PyPI web UI to mark it as withdrawn without breaking the filename namespace.

The dashboard says "stop_requested: false" forever. The polling is 2-second intervals. Click Stop again — if the run is mid-round (a 60-second LLM call), the soft-stop flag only takes effect at the next round boundary.

Can I run multiple experiments in parallel from the same machine? Yes — but each one needs its own --name and --output directory to avoid colliding. ASI-Evolve's database and cognition store will overwrite each other if they share a directory.

License

• LICENSE — MIT for the wrapper code (everything outside src/asi_evolve/_vendor/).
• LICENSE-VENDORED — Apache 2.0 for the upstream ASI-Evolve code.
• NOTICE — Citation requirements + third-party dependency credits.

Both licenses ship in the wheel. Default install respects both.

Acknowledgements

• GAIR-NLP for the ASI-Evolve framework and the paper.
• MiniMax for the underlying model that powers the recommended default provider.
• The original open-source community of evolutionary-search researchers who validated the technique on problems we're only beginning to apply it to.