onthefly-ai 0.0.4

Human-in-the-Loop ML Training Orchestrator.

OnTheFly

OnTheFly lets you steer PyTorch training runs live from VS Code — pause, inspect, fork, merge, and export without ever leaving your IDE or sending data to the cloud.

Today, most training is fire-and-forget: you start a long run, wait, and only discover failure cases or weak slices after it finishes. Fixing them means reruns, one-off notebooks, and manual data slicing.

OnTheFly turns that into an interactive loop:

• watch per-sample loss and slices while you train
• pause to inspect hard examples or drift
• fork short-budget specialists on rough regions
• merge improvements back into a single exportable model

All of this runs fully offline in a local VS Code extension — no accounts, no tokens, no external services.

[!IMPORTANT] Project status: Beta. APIs, UI flows, and file formats may change before v1.0. Expect rough edges and please report issues.

---

When should you use OnTheFly?

OnTheFly is aimed at people who:

• train PyTorch models (classification, regression, etc.) and want more visibility than TensorBoard/print logs
• care about bad slices / drift / outliers and don't want to wait until the run is over to investigate
• prefer a local, offline workflow inside VS Code rather than wiring up cloud dashboards

---

Getting Started

Install

pip install onthefly-ai

Requirements

• Python ≥ 3.9
• PyTorch ≥ 2.2 (CUDA 12.x optional)
• OS: Linux, macOS, or Windows
• Visual Studio Code

Sessions & storage

Every session is ephemeral in storage: when a new session begins, the previous session’s storage is cleaned up. Exporting a session is equivalent to saving a session.

Quickstart

import torch, torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset
from onthefly import quickstart

# toy dataset
X = torch.randn(4096, 28*28)
y = (X[:, :50].sum(dim=1) > 0).long()
ds = TensorDataset(X, y)
train = DataLoader(ds, batch_size=128, shuffle=True)
val = DataLoader(ds, batch_size=256)
test = DataLoader(ds, batch_size=256)

# tiny model
model = nn.Sequential(nn.Linear(28*28, 64), nn.ReLU(), nn.Linear(64, 2))
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
loss = nn.CrossEntropyLoss()

quickstart(
    project="mnist-demo",
    run_name="baseline",
    model=model,
    optimizer=opt,
    loss_fn=loss,
    train_loader=train,
    val_loader=val,
    test_loader=test,
    max_epochs=1,
    do_test_after=True,
)

Don't run this just yet — you'll start training from the dashboard controls instead.

Open the VS Code dashboard

1. Open VS Code → Command Palette (Ctrl/Cmd + Shift + P).
2. Select the “On the Fly: Show Dashboard” command.
3. Select your Python interpreter and training script.
4. Press ▶ Run to start/monitor training, inspect clusters, and compare experts.

---

Features

Mid-training control & visibility

• Start and control training from the VS Code dashboard.
• Stream per-sample loss (optionally grad-norm, margin) with robust quantiles to surface loss tails early.
• Run mid-run health check-ups to detect instability and configuration issues before they cascade.

Fork, specialize, and merge

• Fork short-budget specialists from high-loss tails or residual clusters, then route with a lightweight gate.
• Compare experts side-by-side on target slices.
• Merge via SWA, distillation, Fisher Soup, or adapter fusion; view model lineage (parent/children) before committing.

Data & sessions

• One-click export of indices/rows for any slice to CSV / Parquet / JSON.
• Ephemeral sessions: storage is cleared when a new session begins; exporting is how you save.
• Portable sessions: exported sessions include the final model and can be re-imported to run tests, reports, or further training.

Training backend

• Mirrors training in OnTheFly’s backend for any torch.nn.Module (including custom ones) and standard DataLoaders.
• Deterministic distributed runs, with a surfaced determinism “health check” for monitoring.
• Seamless continuation of training after tests, whether they were triggered automatically or manually.

---

Manual human-in-the-loop

Instead of trusting a long run and hoping for the best, you keep tight control over when to pause, inspect, fork, and merge — with deterministic actions and evidence in front of you.

What you can do

• Pause/Resume at any time to take a clean snapshot.
• Inspect before acting: View per-sample loss distributions, export subsets for offline analysis.
• Approve or edit plan cards prior to execution.
• Compare experts on target slices.
• Merge on your terms via SWA / Distill / Fisher Soup / adapter fusion.
• Run health check-ups mid-run to validate determinism, gradients, and metrics before committing to longer budgets.
• Export & import sessions knowing that exported sessions include the final model and can later be imported, tested, and trained further.

Typical manual loop

1. Pause when drift or a weak slice appears.
2. Inspect loss tails, export a subset for a quick notebook check.
3. Fork a short-budget specialist for chosen samples, with desired parameters.
4. Evaluate on target slices; iterate if needed.
5. Merge improvements and resume training.
6. Export the session for traceability, or import a prior session to continue training or generate reports.

Method (at a glance)

Train a generalist, detect hard cases, focus on those specialists, learn a gating network, and export a unified MoE for inference. Or, don't use forking at all; simply manage your model development from VS Code without connecting to any externals or cloud.

1. Train a compact generalist on all data.
2. Hard-sample mining flags high-loss examples online.
3. Clustering groups hard samples into candidate regimes.
4. Boost rough areas of the loss curve by forking specialists.
5. Choose a gating network to unify experts.
6. Benchmark fairly against a monolithic baseline with matched compute.

License

This project is licensed under the MIT License – see the LICENSE file for details.

---

Citation

If you use this project in research, please cite:

@software{onthefly2025,
  title        = {OnTheFly: Human-in-the-Loop ML Orchestrator},
  author       = {Luke Skertich},
  year         = {2025},
  url          = {https://github.com/KSkert/onthefly}
}