wastebook 0.0.2

Automated research lab notebook: hypotheses, code diffs, results, and conclusions for every experiment you run.

Waste Book

An automated research lab notebook. Experiment trackers record what the metrics were; wastebook records why you ran it, what you changed, and what you concluded — automatically.

wb run -- ./launch_seeds.sh --algo sac --env Walker2d-v4

Every launch captures the git state (HEAD + dirty diff as a reproducible patch), the exact command, and a hypothesis — prompted at launch, drafted by an LLM from your diff. Results flow in from your existing tracker (wandb first; wastebook never reimplements logging). Trials group automatically, statistics come with confidence intervals and significance tests, and an LLM pass drafts conclusions and surfaces anomalies in metrics you weren't watching.

See docs/walkthrough.md for a full worked example (PPO vs SAC, sweeps, multi-seed stats, the review flow).

Principles

• wastebook never owns execution. It wraps whatever you already run — a multi-seed launcher script, a slurm submit, a sweep agent. Tracker runs link to the experiment through propagated environment variables; completion is tracked through tracker run states, so detached launchers just work.
• Never reinvent logging. Metrics stay in wandb/tensorboard/etc., behind an adapter protocol.
• The hypothesis is the soul. Capture is low-friction (prompt with an LLM draft, -m flag, backfill later) but always present in the record. And it's never the only record of intent: at every launch the LLM writes its own observation — an independent read of what the launch actually does (exact parameters changed, factors varied, metrics to watch, relation to prior experiments) — stored as an llm-authored note. Your one-liner says what you think; the observation documents what's going on, and both are searchable later (wb search, MCP).
• Local-first. SQLite in .wastebook/ (gitignored); the generated labbook/ markdown is committed so your research history travels with the repo.

Installation

wastebook is on PyPI. The recommended install is as a uv tool, which puts the wb command on your PATH so you can call it directly from any repo — no uv run prefix, no activating environments:

uv tool install "wastebook[wandb]"

wb --version           # works from anywhere
wb run -- python train.py

Upgrade later with uv tool upgrade wastebook.

Alternatively, add it as a dependency of your research project (uv add "wastebook[wandb]") — then it runs as uv run wb or plain wb inside the activated environment. The tool install is the better default: the notebook wraps your project, it doesn't need to live inside it.

Quick start

cd your-research-repo
wb init                       # writes wastebook.toml, creates .wastebook/
wb run -- ./launch_seeds.sh   # hypothesis prompt (LLM-drafted from your diff)
wb log                        # timeline; lazy-syncs tracker state first
wb review                     # accept/edit drafted conclusions, metric proposals
wb compare exp-0001 exp-0002  # bootstrap CIs, Welch/Mann-Whitney, Cohen's d, Holm
wb show exp-0002 --diff       # full record incl. the captured patch
wb search "entropy collapse"  # months-later recall
wb book                       # regenerate committed labbook/ markdown
wb mcp                        # serve the record to coding agents (MCP)

Commands

init · run (-m hypothesis, -y accept drafts, --group attach-by-label, --pin shadow commit, -e/--experiment reuse-or-create by id, -p/--project) · list · log · show · compare · groups (list/name an experiment's trial groups) · ask (do we have the data to answer this?) · project (list/use/new/show/assign) · review · sync · status (one-screen system summary) · llm (verify the configured model with a live round-trip) · search · book · mcp · annotate · conclude · exclude · merge (fold a mis-grouped experiment into another) · aggregate (fold experiments together) · impact (does a code change reach the launched script?) · link (attach a tracker run that launched outside wb run)

Concepts

The record is a strict hierarchy. From the top:

repository
└── project          one line of experimentation (a research question)
    └── experiment   one hypothesis
        ├── launch   one `wb run` invocation (command + code snapshot)
        └── run      one tracker run (e.g. one wandb run) the launch produced
            └─ grouped into ──┐
        ┌─────────────────────┘
        └── trial group       one distinct configuration within the experiment
            └── trial         one run of that exact configuration (a seed/replicate)

Project — a line of experimentation inside the repo, framed by a research question (e.g. ppo-clip: "does clip range matter?" vs arch-search). Declared in wastebook.toml as [projects.<name>], optionally with its own key metrics. Everything an experiment does stays inside its project: grouping candidates, varies_from lineage, and LLM context never cross projects. A repo that never declares projects has exactly one, implicit default project and behaves as if the layer didn't exist.

Experiment — one hypothesis under test. Created by wb run (with auto-grouping deciding new-vs-attach), belongs to exactly one project, and accumulates launches, runs, findings, notes, and a conclusion. Alongside the hypothesis, the LLM records its own observation of each new experiment — an independent description of what the launch actually changes.

Auto-grouping compares a launch against the whole project, nearest code first — not just the latest experiment. Two identity tiers are checked against every experiment: byte identity (same code + command), then a semantic impact digest — a Merkle-style hash of the AST of every module the entry point can reach — so reformatting, comment edits, and rewrites of unreachable code all still count as the same experiment. Beyond identity, candidates are ranked by impact-weighted code distance: changed lines bucketed into AST blocks (functions/methods/module residue) and weighted by whether that block can affect the launched script — provably-unreachable churn weighs zero, comment-only and moved code weighs ~zero, a changed command flag weighs like code. Run some experiments, spend a week on something else, come back — the returning launch reattaches to the original experiment (its trials land in the right trial groups), and a new experiment's varies_from lineage points at its nearest ancestor, not the unrelated detour.

Launch — one wb run invocation: the exact command plus a code snapshot (git HEAD + dirty patch). An experiment run on three machines has three launches.

Run — one tracker-side run (wandb etc.) discovered via the linking key. One launch of a multi-seed script yields many runs. wastebook never creates runs; it observes them.

Trial group — one distinct configuration within an experiment, derived automatically from the config axes that vary across its runs (a sweep over lr yields one group per lr value; seed-like keys never split groups). The group is the unit statistics compare. Groups are nameable (wb groups exp-0012 --name 1 baseline) and their identity is deterministic (experiment × config axes), so two machines deriving them independently converge to the same records on sync. An experiment whose runs all share one configuration has a single implicit group, shown as (all).

Trial — one run within a trial group: a replicate of that exact configuration. Trials are what give a group its n.

Breaking change (v0.2). This hierarchy replaces the v0.1 flat record: "arms" are now persisted trial-group entities (the runs.arm label column is gone) and every experiment belongs to a project. Pre-0.2 databases and sync records are not migrated — delete .wastebook/wastebook.db (and any record-sync remote) and re-init.

[projects.ppo-clip]
description = "PPO clipping ablations"
question = "Does clip range materially affect final return?"

[[projects.ppo-clip.metrics]]     # optional: replaces the global [[metrics]]
name = "eval/return"              # for this project (wholesale, not merged)
primary = true

$ wb project use ppo-clip        # active project for this machine
$ wb run -p arch-search -- ...   # or per-launch
$ wb groups exp-0012             # the derived trial groups
  #  NAME      LABEL       RUNS  eval/return
  1  —         lr=0.0001      4  96 ± 1.4 (n=4)
  2  baseline  lr=0.001       4  81 ± 1.4 (n=4)
$ wb groups exp-0012 --name 1 "tuned"   # names survive re-syncs & travel

Do we have the data to answer this? (ask)

wb ask (and the MCP tool assess_hypothesis) checks a new question against everything already recorded — every experiment's trial groups and which metrics each recorded — then answers answerable, partial, or no. When the evidence exists it doesn't stop at pointing: it runs the real statistics on the identified groups.

$ wb ask "does a smaller learning rate help final return?"
◆ answerable — exp-0012 sweeps lr with eval/return recorded on both sides (n=4/group)

eval/return · final · n=4 vs n=4
  exp-0012::lr=0.001  (A): 81 ± 1.4 (n=4)
  exp-0012::lr=0.0001 (B): 96 ± 1.4 (n=4)
  Δ = +15   95% bootstrap CI [+13.2, +16.8]
  Welch p=0.0001 · d=10
  → B better on this metric; unlikely to be noise (p=0.0001)

Every group and metric the LLM cites is validated against the record before any statistics run — hallucinated references are dropped, never computed. Without an LLM the command degrades to the data inventory plus keyword matches.

Does this change actually affect my run? (impact)

wastebook builds a static import + call graph from the launch command and checks whether your changed lines can even execute. It's used two ways:

• Grouping is now provable. When a change cannot reach the launched entry point, the new run is grouped as a trial of the prior experiment with certainty — no LLM guess. Reachable changes still get the LLM's does-it-matter judgment.
• wb impact inspects it directly — your uncommitted changes, or an existing experiment's:

$ wb impact --entry train.py
  verdict UNRELATED — (2 unrelated)
  ○ plotting.py · render_curves — symbol is never referenced anywhere reachable
  ○ notebooks/scratch.py — module is never imported or spawned from the entry point

The analysis is sound for the negative: it reports UNRELATED only when it can prove the change can't run. Dynamic imports, getattr, or a subprocess whose target it can't resolve degrade to UNCERTAIN rather than risk a false "unrelated" that would merge two genuinely different experiments. Subprocess launchers (wb run -- ./launch.sh) are followed when the spawned script is a literal path/-m target.

Splitting one experiment across machines

Pass an explicit id to tie separate launches together — even from different machines — without any shared state:

# on your laptop, and again on an EC2 box, and again on a slurm node:
wb run -e pfsp-sweep -- ./train.sh --seed 1
wb run -e pfsp-sweep -- ./train.sh --seed 2

The id becomes the tracker linking key, so every run stamped pfsp-sweep aggregates into one experiment on whichever machine syncs — no git, no shared DB. Reusing an existing id attaches another launch (the hypothesis is kept). To reunite experiments that were created separately, merge them after the fact with wb aggregate <target> <source>….

Syncing the notebook across workstations

The record — hypotheses, code snapshots (with the captured patch), launches, runs, metrics, findings, notes, conclusions — lives in a local SQLite database, so wastebook works fully offline. Point [sync] at a remote and that record propagates to every workstation, no git commits from throwaway machines required:

[sync]
remote = "s3://my-lab/wastebook"   # "wandb", or any fsspec url (s3://, gs://, hf://)
# auto = false                   # opt out of pull-before-read / push-after-write

• remote = "wandb" rides on the wandb project you already use — the record travels as a versioned wastebook-record artifact. No extra credentials.
• An object store (s3://…, gs://…, hf://datasets/user/repo, or a local path) needs the matching driver — uv tool install "wastebook[s3]" (or gcs, hf) — and reads credentials from the usual provider environment chains (AWS_*, GOOGLE_APPLICATION_CREDENTIALS, HF_TOKEN). wastebook never stores keys.

With auto on (the default), reads pull the latest first and record-producing commands push when they finish; wb sync always reconciles both directions. Merging is by stable UUID with last-writer-wins on each experiment's own fields and a union of its launches/runs/notes, so concurrent work on different machines converges without clobbering. A -e id gets the same UUID on every machine, so those launches merge into a single experiment on sync. The local database stays the source of truth: a missing or misconfigured remote degrades to a warning, never a blocked command.

Artifact experiments (generative models, VLM judging)

Not every experiment logs metrics — sometimes the result is a file: images or videos from a generation API, rendered outputs, model responses. wastebook handles these with watched-directory capture and a VLM rubric:

[adapters.artifacts]
watch = ["outputs/"]          # where your script writes/downloads results

[[criteria]]                  # the rubric — criteria play the role of metrics
name = "prompt_adherence"
description = "How faithfully does the image depict the requested prompt?"
scale = 10

[[criteria]]
name = "anatomical_quality"
description = "Hands, faces, limbs are plausible; no extra fingers."
scale = 10
primary = true

$ wb run -m "cfg=4.5 fixes hand artifacts" -- python generate.py
  ↳ wrapper exited (0)
  ↻ artifacts: 8 new file(s) under outputs/ → exp-0003

$ wb sync
  ↻ exp-0003: judged 8 artifact(s) × 2 criteria
  ◈ artifacts: 2/8 images still show fused fingers

$ wb compare exp-0002 exp-0003
  anatomical_quality · n=8 vs n=8
    Δ = +2.1  [95% CI: +0.7, +3.5] · Welch p=0.012

How it works: files that appear under the watched dirs during a launch are linked to the experiment (use wastebook.artifact(path) in your script for outputs elsewhere). Each artifact is recorded by reference + sha256 — files stay where your script put them, and scores/observations survive even if outputs are cleaned. At sync, a vision-capable LLM scores every artifact against the rubric; each artifact counts as one sample, so the full statistics pipeline (CIs, significance, compare) works with n = number of generations. Per-artifact observations aggregate into findings, and judgments are cached by content hash — reopening an experiment never re-judges old artifacts. Videos are judged from evenly spaced frames (requires ffmpeg; captured-but-not-judged without it).

LLM configuration

The LLM drafts hypotheses from your diffs, summarizes changes, groups runs semantically, narrates discovery findings, and drafts conclusions. It is configured in wastebook.toml (any litellm model string) with credentials from the environment — wastebook never stores keys:

[llm]
model = "anthropic/claude-sonnet-4-6"
context_budget = 12000     # max tokens of experiment history per call

export ANTHROPIC_API_KEY=sk-ant-...   # or OPENAI_API_KEY, GEMINI_API_KEY, LITELLM_API_KEY

Local models via ollama — no key needed; just point at a local model:

[llm]
model = "ollama/llama3.1"
# api_base = "http://gpu-box:11434"   # only if ollama isn't on localhost:11434
# autostart_ollama = false            # opt out of launching the daemon for you

If the model is a local ollama one and the daemon isn't running, wastebook starts it on first use (bounded wait, then degrades if it can't) — no ollama serve by hand. It won't pull missing weights, though: run ollama pull <model> once. Remote hosts (api_base elsewhere) are left untouched.

api_base also works for any OpenAI-compatible server (vLLM, LM Studio):

[llm]
model = "hosted_vllm/meta-llama/Llama-3.1-70B-Instruct"
api_base = "http://localhost:8000/v1"

Without any of these, every LLM feature degrades gracefully: launches never block, raw diffs are stored instead of summaries, grouping falls back to byte-identity, and conclusions are written by hand in wb review. Set enabled = false under [llm] to turn the LLM off explicitly, and wb exclude exp-NNNN to keep any experiment out of LLM context.

Development

The project is managed with uv. From a checkout:

uv sync                 # creates .venv from uv.lock (includes the dev group)
uv run pytest           # run the test suite
uv run wb --help   # run the CLI from source

Extras: uv sync --extra wandb --extra mcp to develop against the adapters.

Releases go to PyPI with:

uv build                # sdist + wheel into dist/
uv publish              # needs a PyPI token (UV_PUBLISH_TOKEN)

Status

v0.2: the full loop works — launch capture (git patch snapshots, optional shadow commits), LLM diff summaries + hypothesis drafts (litellm; degrades gracefully without a key), wandb ingestion with adapter-state-driven lazy sync, semantic auto-grouping (with static code-reachability), projects namespacing separate lines of experimentation, trial groups auto-derived from run configs as first-class nameable entities, the statistics toolkit, the discovery pass over all logged series, evidence assessment for new hypotheses (ask / MCP assess_hypothesis), drafted conclusions with a review flow, committed labbook/ rendering, search, an MCP server, and remote record sync (wandb or any object store) so the notebook travels across workstations. Planned next: tensorboard/mlflow adapters.