runwise 0.4.0

Token-efficient ML training run analysis for AI agents

Runwise

Token-efficient ML training run analysis for AI agents.

Runwise parses W&B and local training logs, generating condensed summaries optimized for LLM context windows. Designed for collaborative debugging between humans and AI assistants.

Features

• Token-efficient output: Summaries designed to fit in LLM context without wasting tokens
• Sparkline visualizations: Unicode trend graphs in ~10 tokens (e.g., loss: ▇▆▅▃▂▁ ↓)
• Anomaly detection: Automatically flags spikes, overfitting, plateaus, gradient issues
• W&B integration: Parse local wandb directories or use cloud API
• W&B API support: Access remote runs without local sync (requires wandb package)
• TensorBoard support: Optional parsing of tfevents files (requires tensorboard package)
• Local log support: JSONL and other common log formats
• Markdown export: --format md flag for GitHub issues, Notion, and documentation
• Configurable schemas: Define metrics relevant to your project
• MCP server: Direct integration with Claude Code and other MCP-compatible assistants
• CLI tool: Quick analysis from the command line

Installation

pip install runwise

# Or from source:
git clone https://github.com/jasegehring/runwise
cd runwise
pip install -e .

Quick Start

CLI Usage

# List recent runs with sparkline trends
runwise list
runwise list --no-spark                        # Without sparklines (faster)

# Analyze latest run (includes anomaly detection + sparklines)
runwise latest
runwise latest -k loss,val_loss,accuracy       # Show only specific metrics
runwise latest --no-anomalies                  # Skip anomaly detection
runwise latest --no-spark                      # Skip sparklines

# Analyze specific run
runwise run abc123xyz
runwise run abc123xyz --no-spark --no-anomalies

# Compare two runs
runwise compare run_a run_b
runwise compare run_a run_b -f val             # Filter to validation metrics only
runwise compare run_a run_b -g                 # Group by metric prefix (train/, val/)
runwise compare run_a run_b -t 5               # Only show >5% changes
runwise compare run_a run_b -d                 # Show config differences

# Step-matched comparison (essential for curriculum learning)
runwise compare run_a@50000 run_b@50000        # Compare at same training step
runwise compare run_a@10000 run_b@20000        # Compare at different steps
runwise compare run_a@50000 run_b -f val       # Step-matched, validation only

# Show hyperparameters/config
runwise config                                 # Latest run
runwise config abc123                          # Specific run

# Show run context (name, notes, tags)
runwise notes                                  # Latest run
runwise notes abc123                           # Specific run

# Find best run by metric
runwise best val_loss                          # Lower is better (default)
runwise best accuracy --max                    # Higher is better
runwise best val_loss -n 20                    # Consider last 20 runs

# Get downsampled history (handles million-step runs efficiently)
runwise history                               # Auto-detects common metrics
runwise history -k loss,val_loss              # Specify keys explicitly
runwise history abc123 -k loss -n 100         # Specific run, 100 samples

# Get history statistics (even more compact)
runwise stats                                 # Auto-detects common metrics
runwise stats -k loss,val_loss,grad_norm      # Specify keys explicitly

# Stability analysis (rolling standard deviation)
runwise stability                             # Analyze training stability
runwise stability -k loss,val_loss            # Specify metrics
runwise stability -w 50                       # Custom window size (default: 100)
runwise stability --csv                       # Output as CSV

# List available metric keys
runwise keys                                   # Latest run
runwise keys abc123                            # Specific run

# Live training status
runwise live

# Local JSONL logs (standalone files, not W&B)
runwise local                                  # List logs in logs/ directory
runwise local metrics.jsonl --keys             # List available metric keys
runwise local metrics.jsonl --history -k loss,val_loss  # Get history CSV
runwise local metrics.jsonl --stats -k loss,val_loss    # Get statistics
runwise local metrics.jsonl --stability -k loss         # Stability analysis

# TensorBoard support (requires: pip install tensorboard)
runwise tb                                     # List TB runs
runwise tb -r train_1                          # Summarize specific TB run

# W&B API support (requires: pip install wandb)
runwise api -p my-project                      # List runs (auto-detects project from wandb/)
runwise api -p my-project -r abc123            # Summarize specific run
runwise api -p my-project -r abc123 -k loss,val_loss  # Show specific metrics
runwise api -p my-project --best val/accuracy --max   # Find best run by metric
runwise api -p my-project -r run_id --history -k loss # Get metric history
runwise api -p my-project -c run1,run2         # Compare two runs
runwise api -p my-project -c run1,run2 -f val -t 5    # Compare with filtering
runwise api -p my-project --state running      # Filter by state

# Markdown output (for GitHub issues, Notion, etc.)
runwise latest --format md                     # Markdown summary
runwise list --format md                       # Markdown table
runwise compare run_a run_b --format md        # Markdown comparison

Python API

from runwise import RunAnalyzer, RunwiseConfig

# Auto-detect configuration from project
analyzer = RunAnalyzer()

# List runs
runs = analyzer.list_runs(limit=10)
print(analyzer.format_run_list(runs))

# Analyze latest
run = analyzer.get_latest_run()
print(analyzer.summarize_run(run))

# Compare runs
run_a = analyzer.find_run("abc123")
run_b = analyzer.find_run("def456")
print(analyzer.compare_runs(run_a, run_b))

# Get hyperparameters/config
print(analyzer.get_config(run))

# Find best run by metric
best, ranked = analyzer.find_best_run("val_loss", limit=10, higher_is_better=False)
print(analyzer.format_best_run("val_loss"))

# Get downsampled history (efficient for large runs)
history_csv = analyzer.get_history(run, ["loss", "val_loss"], samples=500)
print(history_csv)

# Get statistics only (most token-efficient)
stats = analyzer.get_history_stats(run, ["loss", "val_loss"])
print(stats)

# List available keys when you don't know what's logged
print(analyzer.list_available_keys(run))

MCP Server (Claude Code Integration)

Add to your Claude Code MCP settings (~/.claude/settings.json):

{
    "mcpServers": {
        "runwise": {
            "command": "python",
            "args": ["-m", "runwise.mcp_server"],
            "env": {
                "RUNWISE_PROJECT_ROOT": "/path/to/your/project"
            }
        }
    }
}

Then in Claude Code, you can ask:

• "Show me the latest training run"
• "Compare runs abc and def"
• "What's the live training status?"

Configuration

Create a runwise.json in your project root:

{
    "project_name": "My ML Project",
    "wandb_dir": "wandb",
    "logs_dir": "logs",
    "downsample_interval": 1000,
    "schema_inline": {
        "loss_key": "train/loss",
        "step_key": "_step",
        "primary_metric": "train/accuracy",
        "primary_metric_name": "Accuracy",
        "validation_sets": {
            "val": "Validation",
            "test": "Test"
        },
        "groups": [
            {
                "name": "training",
                "display_name": "TRAINING",
                "metrics": {
                    "train/loss": {"display": "Loss", "format": ".4f", "higher_is_better": false},
                    "train/accuracy": {"display": "Accuracy", "format": ".1%", "higher_is_better": true}
                }
            }
        ]
    },
    "anomaly_detection": {
        "spike_threshold": 3.5,
        "overfit_ratio_threshold": 1.5,
        "plateau_improvement_threshold": 0.01
    }
}

Anomaly Detection Thresholds

Customize anomaly sensitivity for your domain:

Setting	Default	Description
spike_threshold	3.5	MAD score threshold for spike detection (lower = more sensitive)
spike_window	100	Rolling window size for spike detection
overfit_ratio_threshold	1.5	Val/train ratio increase to flag overfitting (1.5 = 50% increase)
overfit_baseline_steps	[100, 500]	Step range for calculating baseline ratio
plateau_min_steps	500	Minimum steps before checking for plateau
plateau_improvement_threshold	0.01	Required improvement (0.01 = 1%)
gradient_vanish_threshold	1e-7	Gradient norm below this = vanishing
gradient_explode_multiplier	10.0	Gradient above 10x mean = exploding
throughput_drop_threshold	0.4	Throughput drop of 40% = system issue

Example: RL training (expect more variance)

{
    "anomaly_detection": {
        "spike_threshold": 5.0,
        "plateau_improvement_threshold": 0.005
    }
}

Example: Fine-tuning (strict monitoring)

{
    "anomaly_detection": {
        "spike_threshold": 2.0,
        "overfit_ratio_threshold": 1.2
    }
}

Or initialize with defaults:

runwise init --name "My Project"

W&B Best Practices

To ensure Runwise can read your training history, follow these W&B best practices:

Always Call wandb.finish()

The wandb-history.jsonl file (which stores training metrics) may not be fully written if your run is killed or crashes. Always ensure proper cleanup:

# Option 1: Context manager (recommended)
with wandb.init(project="my-project") as run:
    for step in range(1000):
        wandb.log({"loss": loss, "accuracy": acc})
    # wandb.finish() called automatically

# Option 2: Explicit finish
run = wandb.init(project="my-project")
try:
    for step in range(1000):
        wandb.log({"loss": loss, "accuracy": acc})
finally:
    wandb.finish()  # Always called, even on error

Sync Killed Runs

If a run was killed (Ctrl+C, OOM, etc.) before wandb.finish() was called:

# Sync a specific run directory
wandb sync wandb/run-20251214_120000-abc123xyz

# Sync all unsynced runs
wandb sync --sync-all

Troubleshooting Missing History

If runwise history shows "No history data found":

1. Check the run directory: Look for wandb-history.jsonl in wandb/run-*/files/
2. Use alternatives: runwise stats uses wandb-summary.json (final values only)
3. Check run state: Killed/crashed runs may have incomplete data

Example Output

Run List (with sparkline trends)

RECENT RUNS (My ML Project):

ID           State     Date            Steps     Accuracy        Trend
------------------------------------------------------------------------
xyz789       RUNNING   2025-12-14      5,000       82.3%    ▇▆▅▄▃▂▁▁↓
abc123       FINISHED  2025-12-14     50,000       95.2%    ▇▅▃▂▁▁▁▁↓
def456       CRASHED   2025-12-13      2,341       45.0%    ▁▁▂▅▇▇▇▇↑

Run Summary (with anomaly detection)

=== My ML Project Run Summary ===
Run: abc123xyz | Step: 50,000 | Runtime: 12.5h

ANOMALIES:
  ! Overfitting: val/train ratio +35% vs baseline

TRAINING:
  Loss: 0.2341  ▇▆▅▄▃▂▂▁▁▁
  Accuracy: 87.3%  ▁▂▃▄▅▆▇▇▇█

VALIDATION:
  Validation: 85.2%
  Test: 83.7%

Config/Hyperparameters

CONFIG: abc123

  batch_size: 64
  dropout: 0.15
  learning_rate: 5.00e-04
  model: transformer
  num_layers: 8

Best Run

BEST RUN BY val_loss (from last 10 runs):

  BEST: abc123 = 0.1234

Rank   Run ID       State            val_loss
---------------------------------------------
1      abc123       finished           0.1234 *
2      def456       finished           0.1567
3      ghi789       crashed            0.8901

Stability Analysis

STABILITY ANALYSIS: abc123
Window size: 100 steps | Total: 50,000 steps

Metric           Slope/1k  Trend    Avg Std  Final Std     Status
--------------------------------------------------------------------
loss             -0.0023      ↓     0.0234     0.0012     STABLE
val_loss         -0.0015      →     0.0456     0.0089   MODERATE
grad_norm        +0.0089      ↑     0.3210     0.4521      NOISY

Slope/1k: change per 1000 steps (robust Theil-Sen estimate)
Trend: ↓=stabilizing  →=steady  ↑=destabilizing (variance trend)

Why "Token-Efficient"?

When collaborating with AI assistants on ML debugging, you often need to share training logs. Raw W&B exports or verbose logs can consume thousands of tokens, leaving less context for actual analysis.

Runwise generates summaries that:

• Prioritize actionable metrics: Loss, accuracy, per-step breakdowns
• Detect anomalies: Plateaus, regressions, divergence
• Use compact formatting: Tables, aligned columns, minimal whitespace
• Skip redundant data: No repeated headers, timestamps, or metadata

Handling Large Runs

The history command implements intelligent downsampling:

# A 1,000,000 step run returns exactly 500 data points
runwise history -k loss,val_loss -n 500

How it works:

1. First pass: Count total lines (fast, no parsing)
2. Calculate evenly-spaced sample indices
3. Second pass: Parse only the sampled lines
4. Output as CSV (most token-dense format)

This means a 10GB log file with millions of steps produces ~3000 tokens of output, regardless of size. The LLM never sees the raw file.

Contributing

Contributions welcome! Areas of interest:

• Additional log format parsers (MLflow, etc.)
• W&B API support for remote runs
• GitHub Action for PR comments
• Integration with other AI assistants

License

MIT