labcoat 0.1.2

Automated research lab notebook: hypotheses, code diffs, results, and conclusions for every experiment you run.

labcoat

An automated research lab notebook. Experiment trackers record what the metrics were; labcoat records why you ran it, what you changed, and what you concluded — automatically.

labcoat run -- ./launch_seeds.sh --algo sac --env Walker2d-v4

Every launch captures the git state (HEAD + dirty diff as a reproducible patch), the exact command, and a hypothesis — prompted at launch, drafted by an LLM from your diff. Results flow in from your existing tracker (wandb first; labcoat never reimplements logging). Trials group automatically, statistics come with confidence intervals and significance tests, and an LLM pass drafts conclusions and surfaces anomalies in metrics you weren't watching.

See docs/walkthrough.md for a full worked example (PPO vs SAC, sweeps, multi-seed stats, the review flow).

Principles

• labcoat never owns execution. It wraps whatever you already run — a multi-seed launcher script, a slurm submit, a sweep agent. Tracker runs link to the experiment through propagated environment variables; completion is tracked through tracker run states, so detached launchers just work.
• Never reinvent logging. Metrics stay in wandb/tensorboard/etc., behind an adapter protocol.
• The hypothesis is the soul. Capture is low-friction (prompt with an LLM draft, -m flag, backfill later) but always present in the record.
• Local-first. SQLite in .labcoat/ (gitignored); the generated labbook/ markdown is committed so your research history travels with the repo.

Installation

labcoat is on PyPI. The recommended install is as a uv tool, which puts the labcoat command on your PATH so you can call it directly from any repo — no uv run prefix, no activating environments:

uv tool install "labcoat[wandb]"

labcoat --version           # works from anywhere
labcoat run -- python train.py

Upgrade later with uv tool upgrade labcoat.

Alternatively, add it as a dependency of your research project (uv add "labcoat[wandb]") — then it runs as uv run labcoat or plain labcoat inside the activated environment. The tool install is the better default: the notebook wraps your project, it doesn't need to live inside it.

Quick start

cd your-research-repo
labcoat init                       # writes labcoat.toml, creates .labcoat/
labcoat run -- ./launch_seeds.sh   # hypothesis prompt (LLM-drafted from your diff)
labcoat log                        # timeline; lazy-syncs tracker state first
labcoat review                     # accept/edit drafted conclusions, metric proposals
labcoat compare exp-0001 exp-0002  # bootstrap CIs, Welch/Mann-Whitney, Cohen's d, Holm
labcoat show exp-0002 --diff       # full record incl. the captured patch
labcoat search "entropy collapse"  # months-later recall
labcoat book                       # regenerate committed labbook/ markdown
labcoat mcp                        # serve the record to coding agents (MCP)

Commands

init · run (-m hypothesis, -y accept drafts, --group, --pin shadow commit) · log · show · compare · review · sync · search · book · mcp · annotate · conclude · exclude · group · link (attach a tracker run that launched outside labcoat run)

Artifact experiments (generative models, VLM judging)

Not every experiment logs metrics — sometimes the result is a file: images or videos from a generation API, rendered outputs, model responses. labcoat handles these with watched-directory capture and a VLM rubric:

[adapters.artifacts]
watch = ["outputs/"]          # where your script writes/downloads results

[[criteria]]                  # the rubric — criteria play the role of metrics
name = "prompt_adherence"
description = "How faithfully does the image depict the requested prompt?"
scale = 10

[[criteria]]
name = "anatomical_quality"
description = "Hands, faces, limbs are plausible; no extra fingers."
scale = 10
primary = true

$ labcoat run -m "cfg=4.5 fixes hand artifacts" -- python generate.py
  ↳ wrapper exited (0)
  ↻ artifacts: 8 new file(s) under outputs/ → exp-0003

$ labcoat sync
  ↻ exp-0003: judged 8 artifact(s) × 2 criteria
  ◈ artifacts: 2/8 images still show fused fingers

$ labcoat compare exp-0002 exp-0003
  anatomical_quality · n=8 vs n=8
    Δ = +2.1  [95% CI: +0.7, +3.5] · Welch p=0.012

How it works: files that appear under the watched dirs during a launch are linked to the experiment (use labcoat.artifact(path) in your script for outputs elsewhere). Each artifact is recorded by reference + sha256 — files stay where your script put them, and scores/observations survive even if outputs are cleaned. At sync, a vision-capable LLM scores every artifact against the rubric; each artifact counts as one sample, so the full statistics pipeline (CIs, significance, compare) works with n = number of generations. Per-artifact observations aggregate into findings, and judgments are cached by content hash — reopening an experiment never re-judges old artifacts. Videos are judged from evenly spaced frames (requires ffmpeg; captured-but-not-judged without it).

LLM configuration

The LLM drafts hypotheses from your diffs, summarizes changes, groups runs semantically, narrates discovery findings, and drafts conclusions. It is configured in labcoat.toml (any litellm model string) with credentials from the environment — labcoat never stores keys:

[llm]
model = "anthropic/claude-sonnet-4-6"
context_budget = 12000     # max tokens of experiment history per call

export ANTHROPIC_API_KEY=sk-ant-...   # or OPENAI_API_KEY, GEMINI_API_KEY, LITELLM_API_KEY

Local models via ollama — no key needed; just point at a local model:

[llm]
model = "ollama/llama3.1"
# api_base = "http://gpu-box:11434"   # only if ollama isn't on localhost:11434

api_base also works for any OpenAI-compatible server (vLLM, LM Studio):

[llm]
model = "hosted_vllm/meta-llama/Llama-3.1-70B-Instruct"
api_base = "http://localhost:8000/v1"

Without any of these, every LLM feature degrades gracefully: launches never block, raw diffs are stored instead of summaries, grouping falls back to byte-identity, and conclusions are written by hand in labcoat review. Set enabled = false under [llm] to turn the LLM off explicitly, and labcoat exclude exp-NNNN to keep any experiment out of LLM context.

Development

The project is managed with uv. From a checkout:

uv sync                 # creates .venv from uv.lock (includes the dev group)
uv run pytest           # run the test suite
uv run labcoat --help   # run the CLI from source

Extras: uv sync --extra wandb --extra mcp to develop against the adapters.

Releases go to PyPI with:

uv build                # sdist + wheel into dist/
uv publish              # needs a PyPI token (UV_PUBLISH_TOKEN)

Status

v0.1: the full loop works — launch capture (git patch snapshots, optional shadow commits), LLM diff summaries + hypothesis drafts (litellm; degrades gracefully without a key), wandb ingestion with adapter-state-driven lazy sync, semantic auto-grouping, sweep arms auto-derived from run configs, the statistics toolkit, the discovery pass over all logged series, drafted conclusions with a review flow, committed labbook/ rendering, search, and an MCP server. Planned next: tensorboard/mlflow adapters and remote sync of the record.