The Problem

Manual ML development is a grind: tweak hyperparameters, edit code, run training, check results, repeat. AutoML tools like Optuna search parameter spaces, but they can't invent new architectures, combine techniques creatively, or learn from what failed last time.

The Solution

autoMIL gives coding agents the infrastructure to run experiments autonomously:

Real result: On ovarian cancer HRD prediction, autoMIL autonomously ran 189 experiments and improved the composite score from 0.814 to 0.851 (+4.5%), discovering techniques like R-Drop, focal loss, gradient clipping, and coordinate positional encoding that human researchers hadn't tried.

Key Features

Quick Start

1. Install

# Install as a global CLI tool (recommended)
uv tool install git+https://github.com/leoyin1127/autoMIL.git

# Or install from a local clone
git clone https://github.com/leoyin1127/autoMIL.git
cd autoMIL
uv tool install -e .

Note: If you haven't installed automil globally via uv tool install, prefix all commands with uv run (e.g., uv run automil init). This applies when developing within the autoMIL repo itself.

2. Initialize in your project

cd /path/to/your/project    # any existing git repo
automil init                 # creates automil/ subdirectory

3. Setup

claude
# Type: /automil-setup

automil init --runtime codex                   # or opencode, deepseek-via-opencode, deepseek-via-codex
automil show-skill --runtime codex             # render merged per-runtime skill to stdout

run:
  script: "train.py"          # your training script (any name)

files:
  editable: ["train.py", "models/", "losses/*.py"]   # files, dirs, or globs
  readonly: ["evaluate.py"]                          # what must not change

baseline:
  composite: 0.814             # your starting performance

env:
  required: []                 # vars that MUST be set before submit
  passthrough: [AUTOMIL_*]     # vars forwarded to experiment subprocesses

scoring:
  # Documentation-only. Your training script computes the composite scalar
  # and writes it to result.json directly; the framework does NOT evaluate
  # this string. State the formula here so collaborators can read the
  # recipe at a glance. Examples:
  #   formula: "accuracy"                                          # sklearn-iris
  #   formula: "(val_auc + val_bacc + test_auc + test_bacc) / 4"   # autobench
  formula: ""

cap:
  # Consumer-supplied — 6h is just the autoMIL-paper default. Durations accept
  # 6h / 30m / 90s / 2d (or a bare number = seconds; legacy *_seconds ints still
  # work). Set it the easy way: `automil budget set 6h` / `automil budget show`.
  budget: 6h
  safety_buffer: 30m
  # mode: agent_active (default) bills only while the agent is actually working
  # (PostToolUse activity) — the clock PAUSES while experiments run and the agent
  # waits, so you get more proposing time per budget. wall_clock = legacy
  # continuous now-since-creation. idle_grace_seconds: how long after the agent's
  # last action the clock keeps running (agent_active only).
  mode: agent_active
  idle_grace_seconds: 300

automil check

4. Run

Use tmux to keep the orchestrator and agent running in the background:

# Terminal 1: orchestrator (must stay running)
tmux new -s orchestrator
automil orchestrator start
# Ctrl-b d to detach

# Terminal 2: visualization (optional)
tmux new -s viz
automil viz start            # dashboard at localhost:8420
# Ctrl-b d to detach

# Terminal 3: agent loop
tmux new -s automil
claude --dangerously-skip-permissions   # autonomous mode, no permission prompts
# Type: /automil

5. Watch

automil status               # quick summary
automil rank                 # top proposals
# Open http://localhost:8420  # 3D experiment tree

How It Works

  Your Project (unchanged)          autoMIL Overlay
  ========================          ===============
  src/models/clam.py           -->  automil/config.yaml
  src/train.py                      automil/program.md
  src/data_loader.py                automil/learnings.md
  ...                               automil/orchestrator/
                                       queue/ -> running/ -> archive/
                                       completed/

The experiment cycle:

Agent designs experiment
    |
    v
automil submit --files train.py models/clam.py
    |  (snapshots only changed files)
    v
Orchestrator picks up from queue
    |  (creates git worktree at base commit)
    |  (overlays changed files on top)
    v
Runs on GPU in isolation
    |  (CUDA_VISIBLE_DEVICES masked)
    v
Collects result.json
    |  (Pareto dominance: keep or discard?)
    v
Updates experiment graph
    |  (UCB scoring across branches)
    v
Agent reads results + learnings --> designs next experiment

Each experiment stores only its diff, not the full repo. A worktree provides the complete project context at runtime.

Training Script Contract

The seam between autoMIL and your code is the training-script contract: write a result.json matching automil/schemas/result.schema.json before exiting, honor SIGTERM for partial flush, declare required env vars in automil/config.yaml: env.required. Any language, any ML library qualifies.

The minimum valid payload is:

{"composite": 0.912}

composite is the single scalar the experiment tree uses for ranking (higher is always better; for loss minimization, negate). Everything else is optional. A full example payload from the autobench consumer:

{
  "status": "completed",
  "metrics": {
    "val_auc": 0.870,
    "val_bacc": 0.810,
    "test_auc": 0.872,
    "test_bacc": 0.830
  },
  "composite": 0.851,
  "elapsed_seconds": 4098,
  "peak_vram_mb": 4500
}

The sklearn-iris consumer writes {"composite": <accuracy>} with no metrics dict; both shapes validate against the same schema.

The schema is JSON Schema 2020-12 and is validated at ingest by the orchestrator; malformed payloads transition the node to crashed with a schema-location pointer.

CLI Reference

# Project setup + validation
automil init [--runtime <r>] [--no-healthcheck]   Overlay automil/ on current repo
automil check                                     Validate setup (protected files, env.required, backend, registry)
automil show-skill --runtime <r>                  Render merged per-runtime skill file to stdout

# Experiment lifecycle
automil submit --node <id> --desc "..." [--files <f>] [--max-time SEC]
               [--budget-seconds N] [--safety-buffer-seconds M]
                                                  Snapshot changed files and queue.
                                                  --budget-seconds / --safety-buffer-seconds override
                                                  cap.* for the cell this submit creates (D-134;
                                                  ignored on subsequent submits joining the same cell).
automil cancel <node_id>                          Cancel a running experiment
automil resubmit <node_id>                        Re-queue a terminal experiment as a new node
automil rank                                      Show top-ranked proposals (UCB)
automil propose --parent <id> --kind <k> --desc "..."  Add a proposal (kind: architecture|regularization|hp|data|ensemble)
automil portfolio [--threshold 0.5]               Architecture-vs-HP mix of pending proposals; exits non-zero below target
automil reconcile [--recompute-best]              Sync graph with orchestrator state
automil status                                    Show experiment summary

# Variant registry (Phase 1)
automil port-variant <node_id>                    Convert a node's overlay into a registered variant module
automil promote-variant <variant_id>              Move a gate-passing candidate to canonical
automil refresh-registry                          Regenerate per-kind variants/__init__.py deterministically
automil apply <node_id>                           Apply a node's variant selection to config.yaml
automil revert-baseline                           Reset registry.protected paths to base_commit (mandatory pre-stash)
automil verify-repro <node_id>                    Reproduce a node via the registry path; assert |actual - expected| < tolerance

# Cell budget cap (Phase 4)
automil cell list / status / show <id>            Inspect cell budget state and consumed seconds

# Generalization gate (Phase 5)
automil nominate <node_id>                        Mark keep-status node as a gate candidate
automil promote <candidate_id>                    Run Stage B gate (paired Wilcoxon + bootstrap CI + Bonferroni)
automil gate manifest / status                    Manage / inspect the gate manifest

# Trajectory recorder (Phase 3)
automil trajectory record / export / status       JSONL trajectory capture and redacted export bundle

# Loop + daemons
automil start-loop / stop-loop                    Control agent loop flag
automil orchestrator start / stop / status        GPU scheduler daemon (best-fit bin packing)
automil viz start / stop / status                 3D visualization dashboard at localhost:8420

Run automil <command> --help for full flag listings.

Project Structure

your-project/                    # your repo (untouched)
  src/
  models/
  train.py
  ...
  automil/                       # added by automil init
    config.yaml                  # project settings (run, files, env, scoring, cap, gate, backend, hardware)
    program.md                   # agent instructions for the loop
    learnings.md                 # accumulated insights
    graph.json                   # experiment tree (gitignored)
    cells/                       # cell budget state (Phase 4)
    variants/                    # registered variant modules (Phase 1)
      <parent>/                  #   one subdir per registered parent
        <name>.py                #   committed code; selected via config
        __init__.py              #   regenerated by `automil refresh-registry`
    orchestrator/
      queue/                     # pending
      running/<backend>/         # per-backend live job specs (Phase 6)
      archive/                   # permanent record
        node_0001/
          train.py               # only changed files
          spec.json              # experiment spec
          run.log                # stdout/stderr (orchestrator-owned, drained from backend.log_iter)
          result.json            # metrics
          trajectory.jsonl       # agent prompt + tool-call events (Phase 3, gitignored by default)
      completed/                 # notifications

Agent Compatibility

The canonical skill content lives under _shared/; per-runtime directories ship only diffs/overlays. automil show-skill --runtime <r> renders the merged result to stdout. See the Agent Compatibility Guide.

Examples

Documentation

• Getting Started, full setup, configuration, and usage
• Training-Script Contract, the seam between framework and consumer (6 contract items + SIGTERM patterns)
• Agent Compatibility, per-runtime setup, overlay merge model, multi-runtime asset layout
• Implementation Report, v1.0 architecture, design decisions, and the 9-phase refactor that produced it
• CHANGELOG, v1.0 milestone release notes (BREAKING migration paths included)