whirr

Local experiment orchestration. Queue jobs, track metrics, wake up to results.

whirr is a lightweight, self-hosted alternative to experiment tracking tools. No cloud, no accounts, no external dependencies - just SQLite and your filesystem.

Features

• Job Queue - Submit experiments, run them on workers
• Metric Tracking - Log metrics from Python, store as append-only JSONL
• Multi-GPU Support - One worker per GPU, automatic device assignment
• Process Safety - Orphan prevention with PDEATHSIG, graceful shutdown
• Concurrent Access - SQLite WAL mode for safe multi-process access
• Web Dashboard - Interactive UI for viewing runs and comparing metrics
• Reproducibility - Automatic capture of git commit and pip freeze
• Server Mode - Multi-machine orchestration with PostgreSQL backend

Installation

pip install whirr

Optional dependencies:

# For dashboard
pip install whirr[dashboard]

# For server mode (PostgreSQL, HTTP API)
pip install whirr[server]

# For all features
pip install whirr[all]

Or install from source:

git clone https://github.com/syntropy-systems-oss/whirr.git
cd whirr
pip install -e ".[all]"

Quick Start

Initialize a Project

cd your-ml-project
whirr init

This creates a .whirr/ directory with the database and configuration.

Submit Jobs

# Submit a training job
whirr submit --name baseline -- python train.py --lr 0.01

# With tags for organization
whirr submit --name sweep-lr-001 --tags sweep,lr -- python train.py --lr 0.001

Start Workers

# Start a worker (claims and runs jobs from the queue)
whirr worker

# For multi-GPU, start one worker per GPU
CUDA_VISIBLE_DEVICES=0 whirr worker --device 0 &
CUDA_VISIBLE_DEVICES=1 whirr worker --device 1 &

Monitor Progress

# View active jobs
whirr status

# View specific job details
whirr status 1

# Follow job logs
whirr logs 1 --follow

# List completed runs
whirr runs

Cancel Jobs

# Cancel a specific job
whirr cancel 1

# Cancel all queued jobs
whirr cancel --all-queued

Python Library

Track metrics directly from your training scripts:

import whirr

# Initialize a run
run = whirr.init(
    name="my-experiment",
    config={"lr": 0.01, "batch_size": 32},
    tags=["baseline"]
)

# Log metrics during training
for epoch in range(100):
    loss = train_epoch()
    run.log({"loss": loss, "epoch": epoch}, step=epoch)

# Save summary metrics
run.summary({"best_loss": 0.1, "best_epoch": 42})

# Save artifacts
run.save_artifact("model.pt", "best_model.pt")

run.finish()

Or use as a context manager:

with whirr.init(name="my-run") as run:
    run.log({"loss": 0.5})
    # Automatically marked complete or failed

Direct Runs (No Queue)

You can use whirr for tracking without the job queue:

python train.py  # Uses whirr.init() directly
whirr runs       # Shows the run

Server Mode (Multi-Machine)

For distributed setups with multiple GPU machines. See the full setup guide for detailed instructions.

Quick Setup

1. Start the server (on head node):

# Create .env file with configuration
cat > .env << EOF
POSTGRES_PASSWORD=$(openssl rand -base64 32)
WHIRR_DATA_DIR=/srv/whirr
EOF

# Start PostgreSQL + whirr server
docker compose up -d

2. Install workers (on GPU nodes):

# Download the Rust worker (recommended - only ~2MB, ~10MB RAM)
curl -L https://github.com/syntropy-systems-oss/whirr/releases/latest/download/whirr-worker-linux-x86_64 \
  -o /usr/local/bin/whirr-worker
chmod +x /usr/local/bin/whirr-worker

# Or use the Python worker
pip install whirr[server]

3. Start workers:

# Rust worker
whirr-worker --server http://head-node:8080 --data-dir /mnt/shared/whirr --gpu 0

# Or Python worker
whirr worker --server http://head-node:8080 --data-dir /mnt/shared/whirr --gpu 0

4. Submit jobs:

whirr submit --server http://head-node:8080 -- python train.py --lr 0.01

Project Structure

your-project/
├── .whirr/
│   ├── config.yaml      # Project configuration
│   ├── whirr.db         # SQLite database (WAL mode)
│   └── runs/
│       └── run-id/
│           ├── meta.json       # Run metadata
│           ├── config.json     # Run configuration
│           ├── metrics.jsonl   # Logged metrics
│           ├── output.log      # stdout/stderr
│           └── artifacts/      # Saved files

CLI Reference

Documentation

Full documentation is available in the docs/ directory:

• Getting Started - Installation and first steps
• CLI Reference - All commands and options
• Python API - Library reference and examples
• Architecture - How whirr works under the hood

Development

# Install with dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run specific test file
pytest tests/test_db.py -v

License

MIT License - see LICENSE for details.