dojoml 0.0.17

AI-powered autonomous ML research framework — agent runs experiments against frozen evaluation contracts on your existing pipelines.

🥋 Dojo — An AI-powered autonomous ML research framework.

Run controlled, reproducible ML experiments on your existing pipelines and build a memory of what actually works.

---

---

What is Dojo?

You define a domain — a research area pointing at your data with a fixed evaluation contract. An AI agent runs experiments inside that contract: writing training code, calling frozen load_data and evaluate tools, logging metrics, and recording findings as durable knowledge atoms.

Domain (you define)
  ├── Task            — the contract: load_data + evaluate (frozen, AI-generated at setup)
  ├── Workspace       — your repo / pipeline (local path or git url)
  └── Experiments     — agent-created, many per domain
        └── Knowledge atoms — linked across experiments, accumulating over time

The agent owns the training code. The framework owns evaluation. That separation is what makes the metrics trustworthy run-over-run, and what makes it safe to leave the agent unsupervised.

Inspired by Karpathy's autoresearch — prepare.py is frozen, train.py is fair game, program.md is what the human iterates on. Dojo generalises that pattern to any well-defined ML problem class.

---

Status

⚠️ Proof of Concept — under active development. Open source. Single-tenant, local-first, by design.

• Agent: Claude Agent SDK (uses your local claude CLI auth — no API key needed for runs)
• Storage: Local JSON files in .dojo/ — your data stays on your machine
• Tasks supported: RegressionTask (more types to come once regression is solid)

---

Install

The package on PyPI is dojoml; the CLI binary it installs is dojo.

uv tool install dojoml         # recommended — isolated, on your PATH
# or
pipx install dojoml
# or
pip install dojoml

Prerequisites:

• Python 3.13+
• The claude CLI logged in (Claude Code) — Dojo shells out to it; no ANTHROPIC_API_KEY needed for agent runs.

---

Quickstart — dojo onboard

dojo onboard is the recommended entry point. Run it inside an existing Python project — it adds .dojo/ next to your code, reuses your pyproject.toml / requirements.txt for dependencies, and walks you through everything else:

cd path/to/your/python/project
uv tool install dojoml          # one-time
dojo onboard                    # answers a few questions, generates load_data + evaluate, freezes the task
dojo run                        # the agent starts running experiments

That's it. Your research lives at .dojo/ in the project (knowledge, runs, frozen tools), and your code stays where it always was. dojo onboard asks for:

• the agent + tracking + linker backends (sensible defaults — hit enter)
• a domain name + description
• your research goal, target, success criteria (PROGRAM.md)
• where the data lives + how evaluation should work (SETUP.md)

It then runs the AI tool generator, verifies load_data + evaluate against the frozen regression contract, and freezes the task. If the verifier hits a missing import, onboard offers to install it into the workspace venv and retries automatically.

Don't have a project yet? Try a preset

If you just want to see Dojo work end-to-end on a canned dataset:

mkdir housing && cd housing
dojo onboard --preset california_housing   # ready-to-run PROGRAM.md + SETUP.md
dojo run --max-turns 30

The california_housing preset uses sklearn.datasets.fetch_california_housing and pre-installs scikit-learn, pandas, numpy, matplotlib into a fresh venv. More presets coming.

Scripted setup (dojo init)

For CI or non-interactive use where prompts aren't acceptable, the older four-step path is still available:

dojo init --name housing --task-type regression --non-interactive
$EDITOR PROGRAM.md SETUP.md
dojo task setup
dojo run

If the AI keeps generating the wrong adapters on real-world pipelines (unusual pandas multi-indexes, custom dataset APIs, wrapping an existing evaluator), set Opus 4.7 as the tool-generation model:

DOJO_AGENT__TOOL_GENERATION_MODEL=claude-opus-4-7 dojo task setup

Opus is slower (~30–60s vs 15–30s) but noticeably better at translating a messy SETUP.md into correct load_data / evaluate modules. Set it permanently in .dojo/config.yaml under agent.tool_generation_model.

What happens under the hood:

• dojo onboard / dojo init writes .dojo/config.yaml, creates the domain + regression task with expected_metrics = [rmse, r2, mae], scaffolds PROGRAM.md and SETUP.md, and sets current_domain_id.
• dojo task setup reads SETUP.md, asks the AI to generate load_data + evaluate, runs each tool in a sandbox against its ToolContract, and freezes the task. Verification failures tell you which tool failed and why — fix SETUP.md (or the tool code) and re-run. (dojo onboard runs this automatically as its last step.)
• dojo run starts the agent in-process. The agent writes training code; load_data and evaluate stay frozen. The metric dict from evaluate is the only source of truth — complete_experiment rejects metric keys outside the contract, so the agent can't smuggle in custom numbers.

Useful neighbours:

dojo task show               # current task status, tools, frozen?
dojo runs ls                 # recent runs
dojo runs show               # last run's events + cost
dojo program show            # print the live PROGRAM.md
dojo domain use <name>       # switch active domain

Stopping a run

dojo run blocks the foreground until the agent finishes. To stop it early:

• Ctrl-C in the running terminal — the canonical path. The orchestrator is interrupted, the framework asks the backend to summarise any durable findings as knowledge atoms (a small one-shot LLM call), then prints a final cost line. A second Ctrl-C aborts the cleanup immediately.
• dojo stop [run_id] from another terminal — marks the run STOPPED on disk. This does not halt an in-process foreground run (the orchestrator lives inside the other terminal's Python process); use it to recover records left RUNNING after a hard kill, or to stop server-mode runs.

Reviewing what happened

dojo experiments ls          # rank experiments by the primary metric (best first)
dojo experiments best        # show the single best experiment so far
dojo experiments show <id>   # full detail: hypothesis, metrics, code path, errors
dojo runs show               # last run's events + total cost

dojo experiments ls orders by the task's primary_metric and direction (e.g. rmse minimised), so the leader sits on top regardless of run order. The agent's training code is preserved per-experiment in the workspace as __dojo_train_<experiment_id>.py — cat it to reproduce a run by hand.

Artifacts

Each experiment gets a fresh .dojo/domains/{id}/runs/{eid}/artifacts/ directory. The runner passes its path as artifacts_dir to both train() and evaluate().

• evaluate(..., artifacts_dir) writes durable per-run diagnostics — residual plots, calibration curves, error breakdowns. These are produced on every run and are part of the user-defined evaluation contract in SETUP.md.
• train(..., artifacts_dir) writes opportunistic artifacts — model checkpoints (joblib.dump(model, artifacts_dir / "model.pkl")), training curves, feature importances. The agent decides when an artifact is worth keeping; not every run will write here.

Everything written to artifacts_dir is:

1. Copied into the durable Dojo archive at .dojo/artifacts/experiments/{eid}/....
2. Forwarded to the active tracking backend (MlflowTracker.log_artifact uploads to MLflow; FileTracker records a reference; NoopTracker drops it).

Configuration

Create .dojo/config.yaml in your project root:

agent:
  backend: claude      # "stub" (no LLM, deterministic) or "claude"
tracking:
  backend: file        # "file" or "mlflow"

Or override via environment variables (note the double underscore for nested fields):

DOJO_AGENT__BACKEND=claude
DOJO_TRACKING__BACKEND=mlflow

Web UI / HTTP API (optional)

dojo start                   # FastAPI server on http://localhost:8000

The server reads the same .dojo/ your CLI commands write to, so a CLI-started run is visible to the API and vice versa.

Note: the React frontend is not bundled in the PyPI release yet. If you want the web UI, run it from a checkout — see Development below.

Migrating from v0.0.10

If your domain has a v0.0.10 PROGRAM.md with mixed Goal/Dataset/Evaluate content:

1. Create SETUP.md next to PROGRAM.md with the existing ## Dataset and ## Evaluate sections.
2. Trim PROGRAM.md to ## Goal, ## Target, ## Success, ## Notes.
3. Run dojo task setup again — the regression contract is now v4 (train receives artifacts_dir), so any frozen task needs re-verification anyway.

---

Development

Most of the contributor reference lives in CLAUDE.md (architecture, directory map, "how do I add X" recipes, conventions). This section is the minimum to clone and run tests.

Additional prerequisites for the dev path:

• uv
• just
• Node.js 18+ (only if you want to run the web UI)

git clone https://github.com/Garsdal/Dojo.git && cd Dojo
just dev                     # install backend + frontend deps
just test                    # run the test suite
just lint                    # ruff check
just format                  # auto-fix lint + format

For the full server + web UI dev loop:

just run-stub                # API + frontend with the stub agent (no LLM, deterministic)
just run-claude              # API + frontend with the Claude agent

Backend → http://localhost:8000 · Frontend → http://localhost:5173.

Pointers

• CLAUDE.md — architecture, directory map, conventions, recipes.
• docs/MASTER_PLAN.md — vision and the typed-Task design.
• docs/RELEASING.md — release flow.

HTTP API endpoints

Method	Path	Description
POST	/domains	Create a research domain
POST	/domains/{id}/task	Attach a Task (regression today)
POST	/domains/{id}/tools/generate	AI-generate load_data / evaluate from SETUP.md, verify against contract
POST	/domains/{id}/task/freeze	Freeze the task — gated on every required tool's verification
POST	/domains/{id}/workspace/setup	One-time workspace prep (venv + deps)
POST	/agent/run	Start an agent run on a domain (requires a frozen task)
GET	/agent/runs/{id}/events	Live SSE event stream
GET	/experiments?domain_id=	List experiments
GET	/knowledge?domain_id=	List knowledge atoms
GET	/health	Health check