RSIKit

Evolve and evaluate class-based policies in Gymnasium environments.

Install the shared library with pip install rsikit (Python 3.10+). Research algorithms and examples require a clone of this repository. This is an experimental release; APIs may change.

• Policy: the agent interface: reset, act, and close.
• generate: returns a named Policy subclass from the LLM.
• Run: stores policies, scores, and returned artifacts.
• Executor: owns concurrency and timeouts, using a configurable sandbox.
• DockerSandbox: reuses one container across a run, with independent evaluation processes.
• InProcessDockerSandbox: runs policy and environment together inside Docker, avoiding per-action IPC.
• AlphaEvolve: evolutionary search using Slick and Gymnasium feedback.
• ShinkaEvolve: island archives, adaptive model selection, and diff/rewrite/crossover search.
• LineageSearch: diverse approach families, measured refinement and pivots, and stagnation-based completion.

The shared library lives in rsikit/. The four independent search algorithms live under research/ and import shared functionality from rsikit, never from another algorithm. Run research examples from the repository root; the library wheel includes only rsikit.

from pathlib import Path

import gymnasium as gym
from slick import prompts
from slick.providers import OpenRouterAPI

import rsikit.generation as generation
from rsikit import Executor, Run, generate

# Configure Slick once at application startup.
prompts.TEMPLATE_ROOT = Path(generation.__file__).parent / "prompts"
provider = OpenRouterAPI(model="openai/gpt-oss-120b:nitro")

# Inside an async function:
policy = await generate(
    "Balance CartPole-v1. Observation: cart position/velocity, pole angle/angular "
    "velocity. Action 0 pushes left and 1 pushes right. Maximize surviving steps.",
    provider=provider,
)
executor = Executor(concurrency=4)
with gym.make("CartPole-v1", max_episode_steps=500) as environment:
    async with Run.create(name="cartpole-comparison", environment=environment, executor=executor) as run:
        scores = await run.evaluate([policy], seeds=[0, 1, 2])
        print(policy.name, scores[policy.id])
        print(run.path)

The policy name is generated by the LLM. Pass the returned class directly to the run; there is no source/file argument to assemble. Automatic Python export is on by default (export=False disables it). Use run.evaluate(policies) for a group; it returns one mean score per policy ID. See execution, runs, and recovery.

Docker episodes default to a 60-second wall-clock limit per seed. Exceeding it raises a recoverable PolicyTimeout and terminates that episode's workers while other episodes continue. With DockerSandbox, call_timeout separately limits each policy call.

Executor defaults to InProcessDockerSandbox. To configure its episode timeout:

from rsikit import Executor, InProcessDockerSandbox

executor = Executor(sandbox=InProcessDockerSandbox(episode_timeout=60), concurrency=4)

It keeps the restricted Docker container and a fresh process per episode, but lets the policy share memory with the environment. It protects the host rather than hidden environment state or scoring integrity. Its external episode_timeout covers the complete rollout; call_timeout is not applied. Rebuild the sandbox image before using it. See performance measurements.

Setup

uv venv
uv pip install --python .venv/bin/python -e '.[dev]'
.venv/bin/python -B -m examples.cartpole

Runtime dependencies are Gymnasium, NumPy, Slick (slick-ai), Pydantic, SQLModel, cloudpickle, and Rich.

The Docker worker additionally includes SciPy, python-control (control), CVXPY with OSQP/Clarabel/SCS, scikit-learn (sklearn), and CPU-only PyTorch (torch). All generators and repair prompts share scientific-library guidance. Image builds verify Riccati/LQR, convex solvers, regression and Torch autograd; installed versions are recorded in /opt/worker/libraries.json. Rebuild the image after dependency changes. No sibling checkout or slick-bits dependency is required.

Isolated programs

docker build -t rsikit-sandbox:local -f rsikit/sandbox/Dockerfile .
.venv/bin/python -B -m examples.circle_packing
.venv/bin/python -B -m examples.circle_packing /path/to/solution.py

A program exports Solution(Policy). One policy instance persists throughout an episode. Run evaluates both environment and agent inside the container, in separate processes. The lower-level run_program helper still keeps its environment on the host.

See the API guide for task instructions, class contracts, results, and execution limits.

For finite-shoe blackjack with betting and memory across hands, see the blackjack environment. Benchmark completed hands with .venv/bin/python -m examples.blackjack --compare-gym. Run a Blackjack test with automatic leader videos after each generation using .venv/bin/python -m examples.blackjack_train --output runs/blackjack-test.

For daily BTC portfolio allocation with transaction fees and a seven-year training / three-year validation split, see the Bitcoin environment. Use --env Bitcoin with the optimizer examples, or benchmark locally with .venv/bin/python -m examples.bitcoin (100,000 steps/second minimum).

Generate and compare five policies with OpenRouter

uv pip install --python .venv/bin/python -e '.[openrouter]'
docker build -t rsikit-sandbox:local -f rsikit/sandbox/Dockerfile .
export OPENROUTER_API_KEY='your-key'
.venv/bin/python -B -m examples.inner_loop

This makes five real generation calls through Slick's OpenRouterAPI using openai/gpt-oss-120b:nitro. OpenRouter's Nitro routing selects providers for throughput. The example requests random, angle-only, proportional-derivative, full-state-feedback, and predictive-rollout policies.

Each generated Solution class runs in Docker on CartPole-v1, using identical seeds 0–4 and a 500-step cap. The run submits the five policies as one group, with two evaluations active at a time (--concurrency changes this). The terminal shows mean and per-seed scores; higher is better. Policies with failed episodes are reported without a mean score. There is no evolutionary search or repair loop in this example.

Each run saves run.sqlite, automatically exported Python files under exports/, and comparison reports under a new runs/cartpole-comparison-* directory. These are measured results from your run; the example contains no preset scores or fallback policies. To shorten evaluation or choose a new output folder:

.venv/bin/python -B -m examples.inner_loop --seeds 1 2 3 --max-steps 200 --output runs/my-comparison

The chosen output directory must not already exist. Inspect the policy code and report together: the requested strategy is guidance, not a guarantee that the model follows it. Capped returns can tie, and five seeds do not establish general performance. This demo needs the OpenRouter key only on the host for generation.

To retry unfinished evaluations without generating anything, supply the same environment options (--max-steps, --video) as the original run:

.venv/bin/python -B -m examples.inner_loop --resume runs/YOUR_RUN

For videos, the example configures Gymnasium RecordVideo before creating the run. Add --video (the worker image includes rendering dependencies):

uv pip install --python .venv/bin/python -e '.[video]'
.venv/bin/python -B -m examples.inner_loop --video

AlphaEvolve

See the AlphaEvolve guide for the Python API, search controls, and structured generation contracts. Run the CartPole example with an API model:

uv pip install --python .venv/bin/python -e '.[openrouter]'
# Set OPENROUTER_API_KEY first; this command makes paid model calls.
.venv/bin/python -B -m examples.alphaevolve --generations 25 --batch-size 10

Build the Docker worker above before running the search. The default paper variant uses openai/gpt-oss-120b:nitro and overlaps generation with evaluation:

from research.alphaevolve.paper import search

# evaluate_batch returns policy.id -> EvaluationResult with metrics and descriptors.
await search(generator, evaluate_batch, proposals=250, evaluation_batch_size=10)

Run saves policies, scores, and artifacts during evaluation, and exports Python automatically. The optimizer saves its population in population.sqlite, with metric-specific elites in descriptor bins on each island. Lower primary-score candidates remain available when they win another metric or niche. Ten limits the evaluation batch size; the retained population is separate. Rich progress shows each policy's name and description as it is generated, then scores as evaluations finish. A score table follows each evaluation batch; run.log keeps messages and error details. Self-healing allows two model repairs per policy for malformed generation or sandbox policy failures (--max-repairs changes the limit). Successful scores are reused. An exhausted repair budget discards that policy; surviving policies and later generations continue. A generation can return fewer policies than --batch-size. Generation runs four proposals concurrently by default, including their repair calls. Use --generation-concurrency to change this; --concurrency controls sandbox evaluation separately. Set generation concurrency to 1 for sequential requests. The optimizer lives outside the core in research/alphaevolve/paper/. This independently implements the published mechanisms; the paper does not disclose exact database sampling/migration rules, so those choices are documented in the guide. Select --variant original for the historical scalar-feedback baseline or --variant improved for the historical independent-founding baseline. Run names include the variant and experiment.json saves resolved settings. Continue a paper run with its saved population and settings:

.venv/bin/python -B -m examples.alphaevolve --resume runs/YOUR_RUN --generations 25

This adds 25 batches using the saved batch size. Logs/history stay in the same directory. Interrupted proposals are not replayed; older baseline runs have no optimizer checkpoint. See resume details. See the comparison setup. The CLI also saves per-proposal outcomes, ancestry, repair versions, and per-generation island champions and resets in SQLite. See the history schema and island-curve query.

In the improved variant, initial proposals compete on separate islands; the same program cannot found multiple islands just by changing its name. Mutation prompts include each parent's and inspiration policy's per-seed rewards. The CLI supplies this feedback from Run automatically; selection still ranks policies by mean reward.

For LunarLander (continuous controls with wind) and then BipedalWalker (normal terrain):

uv pip install --python .venv/bin/python -e '.[openrouter,box2d]'
docker build -t rsikit-sandbox:local -f rsikit/sandbox/Dockerfile .
.venv/bin/python -B -m examples.alphaevolve --env LunarLander-v3 --seeds 0 1 2
.venv/bin/python -B -m examples.alphaevolve --env BipedalWalker-v3 --seeds 0 1 2

Both default to a budget of 250 proposals. BipedalWalker uses full rewrites by default (--mode diff overrides). Duplicate top-level Solution definitions are rejected and repaired. Each command creates a separate run. The native time limits apply (1,000 steps for LunarLander, 1,600 for BipedalWalker); --max-steps overrides them. --seeds evaluates every policy on the same starting conditions and returns their mean score. The environment wrapper owns observation, action, reward, and termination instructions, used by both generation and execution. The box2d extra builds box2d-py for Python 3.14; uv supplies SWIG during the build. A C++ compiler is required (on macOS, Xcode Command Line Tools).

After a search finishes, record the three highest-scoring saved policies:

.venv/bin/python -B -m examples.replay runs/YOUR_RUN --env LunarLander-v3 --top 3 --seeds 0 1 2

Use --env BipedalWalker-v3 for a walker run. Match any original --max-steps override. No API key or generation is needed. Gymnasium records fresh episodes in Docker; a separate video Run saves their scores and MP4 files and prints each file's path. Open an MP4 with your video player (open /path/to/video.mp4 on macOS). Ranking uses original mean scores and excludes policies with unfinished evaluations. Replay scores are stored separately, so previously cached scores cannot skip recording.

ShinkaEvolve

The previous main implementation is now available in research/shinkaevolve/, using the same environments, Docker executor, Run storage, and video replay:

.venv/bin/python -B -m examples.shinkaevolve --env LunarLander-v3 --generations 10 --batch-size 25 --seeds 0 1 2 3 4
.venv/bin/python -B -m examples.shinkaevolve --env BipedalWalker-v3 --generations 10 --batch-size 25 --seeds 0 1 2 3 4

Use the OpenRouter/Box2D setup above. The default model is openai/gpt-oss-120b:nitro; repeat --model MODEL for adaptive allocation across an ensemble. The CLI saves proposal ancestry, repair revisions, full island populations, migration, reflection, and model allocation history in SQLite. See the ShinkaEvolve guide for mechanisms, optional novelty checks, the Python API, and analysis queries.

EliteSearch

Keep a leaderboard of 10 elites and evaluate 50 new organisms per generation. Candidates are new ideas, focused edits of an elite, or remixes combining multiple elites. Only measured scores decide promotion; the top 10 from old elites plus newcomers survive. Generation and Docker evaluation overlap, with bounded repair and a Rich leaderboard.

.venv/bin/python -B -m examples.elitesearch --env BipedalWalker-v3 \
  --elites 10 --population 50 --generations 20 \
  --generation-concurrency 100 --concurrency 8 --max-repairs 5

See the EliteSearch guide for the operator mix, scoring, saved generations and Python API. This is independent of LineageSearch.

LineageSearch

Generate broad approach families, evaluate their implementations, and expand promising parents within each family until all families stagnate or the attempt budget is spent. MAKER's typed taxonomy adaptation elects the families first, then all initial approaches beneath them, before any policy implementation. Rich displays the tree; decomposition.json saves it. --decomposition-k 3 uses five competing partitions and requires a discriminator vote lead of three. Each family receives exploration time; additional batches favor strong scores and recent confirmed gains. --cull-percent 90 keeps the top 10% globally across survivors plus newly scored candidates, then expands from those survivors again. Families with no survivors retire; culled candidates stay archived but cannot parent later trials. The optimizer records ancestry, hypotheses, failures, and completion reasons in the Run database. Malformed generations and failed policies receive up to two repair calls per policy (--max-repairs changes the limit). Rich progress shows generation, repairs, evaluation, family completion, and score tables; run.log retains the messages.

.venv/bin/python -B -m examples.lineagesearch --env CartPole-v1 --max-attempts 500

Use the OpenRouter and Docker setup above. Search uses seeds 0–4; the final incumbent is evaluated on separate seeds 100–104. See the LineageSearch guide for the API, selection rules, stagnation settings, and stored history.

Checks

.venv/bin/python -B -m unittest discover -s tests -v
.venv/bin/ruff check .
.venv/bin/ruff format --check .

Host and sandbox Python minor versions must match. The Dockerfile defaults to Python 3.14; use --build-arg PYTHON_VERSION=X.Y when building for another version.

Docker checks skip explicitly when Docker or the worker image is unavailable. Research algorithms, examples, and tests are included in the source distribution and excluded from the library wheel. The core supplies policy generation, shared edit validation, execution, environments, run storage, and progress display.