tlog-ml 0.2.0

Lightweight, local-first experiment logger for neural network training — wandb-shaped API, zero deps, terminal + HTML + web viewers

tlog

A lightweight, local-first experiment logger for neural network training. wandb-shaped API, zero dependencies in your training environment, and three clean ways to look at your runs from a SLURM cluster with nothing but a terminal:

viewer	command	when
terminal dashboard	tlog watch	live charts in a tmux pane — the default
live web dashboard	tlog serve	wandb-like browser UI through an SSH/VS Code port-forward
self-contained HTML	tlog export -o report.html	one file with charts + images; preview in VS Code, scp it, share it
custom report	tlog report spec.md	a markdown narrative with live chart/table/image blocks — write it yourself or have a coding agent compose it

Everything is plain append-only JSONL in a run directory: grep-able, rsync-able, crash-safe, no daemon, no cloud, no account.

 ● demo/baseline (da064b) · step 1500 · finished
  loss  eval  training  timing  memory  console

 loss/charb                              0.3158    loss/dino                              0.07182
  1.552 ┤⡧⣼                                        0.3882 ┤⡧⣼
        │⠇⢹⢿⣠⢀                                            │⡇⢹⣶⣀⣀
        │   ⠹⢹⠢⣧⣄⣀                                        │   ⠛⢹⠢⡦⣆⢀
        │      ⠁⠋⠋⠳⢶⢤⡀                                    │       ⠋⠉⠢⣴⣀⣀
        │           ⠘⠙⠦⠦⢴⢄⣀⡀                              │           ⠘⠙⢢⡧⢦⣀⢀
        │                 ⠉⠙⠛⠓⠶⠤⣤⣠⣠⣀⡀                     │                 ⠉⠙⠋⠳⠶⢤⣤⣴⣠⡄⡀
 0.2771 ┤                         ⠉⠙⠉⠛⠋⠓⠲⠚⠴⠖⠤⠤⠦⡦  0.06998 ┤                         ⠁⠙⠉⠋⠋⠑⠳⠒⠲⠶⠤⡶⣦⣦
        10                                 1490           10                                 1490

 loss/ssim                                0.131    loss/total                              0.5004
 0.6492 ┤⡧⣼                                         2.605 ┤⡧⣼
        │⠇⢹⢶⣀⢀                                            │⠇⢹⢶⣀
        │   ⠹⠹⠦⣶⣄⣀                                        │   ⠹⢹⠢⣦⣄⣀
        │       ⠛⠋⠣⣴⣠⡀                                    │       ⠋⠋⠲⣴⢤⣀
        │          ⠈⠘⠙⠲⠦⢤⣀⢀                               │           ⠘⠙⠦⠦⢤⣀⣀
        │                 ⠉⠙⠋⠓⢴⠤⢤⣤⣠⡀⡀                     │                 ⠈⠙⠋⠲⠴⠤⣤⣠⢠⡀⡀
 0.1174 ┤                         ⠁⠉⠉⠛⠊⠛⠲⠖⠴⠒⠤⡴⠦⣤   0.4672 ┤                         ⠉⠉⠉⠛⠊⠛⠲⠖⠴⠖⠤⠤⠦⣦
        10                                 1490           10                                 1490

 ←/→ pages · ↑/↓ scroll · 1-9 cols (auto) · s smooth (0) · l log (off) · q quit

An actual tlog watch frame — braille-canvas charts in a plain tmux pane.

Install

# latest features (multi-run compare, terminal images, custom reports):
pip install "git+https://github.com/philippe-eecs/tlog"

# released to PyPI (currently 0.1.0 — older; the next tag publishes the rest):
pip install tlog-ml          # distribution is tlog-ml; you still `import tlog`

# for development:
git clone https://github.com/philippe-eecs/tlog && cd tlog
pip install -e ".[dev]"

The git+ form is the one to use on a fresh cluster today — it pulls main with everything below, no PyPI release needed. The core has zero dependencies — nothing to conflict with your torch/jax pins. PIL is used opportunistically if present (image encoding, report downscaling); otherwise a pure-stdlib PNG encoder takes over.

Quickstart

import tlog

run = tlog.init(project="vitok", name="vae-L16", config=vars(args))

for step in range(steps):
    ...
    if step % log_freq == 0:
        tlog.log({"loss/total": loss, "training/lr": lr,
                  "timing/mfu_percent": mfu}, step=step)
    if step % eval_freq == 0:
        tlog.log({f"eval/{k}": v for k, v in eval_stats.items()}, step=step)
        tlog.log_images("eval/recon", [orig, recon], step=step)  # torch/np/PIL

tlog.finish()

Then, in another tmux pane:

tlog                          # == tlog watch: live dashboard of the latest run
tlog watch baseline high-lr   # overlay multiple runs (or a project dir) in one TUI
tlog ls                       # table of runs: step, last loss, slurm job, status
tlog tail                     # live captured console output of the latest run
tlog serve                    # web UI on :8585 (VS Code auto-forwards the port)
tlog export run-a run-b -o compare.html    # side-by-side report

Key namespaces (loss/, eval/, timing/, ...) become chart groups / TUI pages automatically.

What gets captured

tlog.init() records, without being asked:

• SLURM: job id, job name, partition, nodelist, array task id, and the actual sbatch script that launched the job (saved as launch.sh)
• git: commit, branch, dirty flag, and a diff.patch of uncommitted changes
• environment: argv, entrypoint, hostname, user, python/torch/CUDA versions, GPU models, world size
• system metrics (background thread, 10s interval): GPU util/mem/temp/power per device via nvidia-smi, CPU%, RAM — shown as their own chart groups
• console: stdout/stderr teed to console.log (tqdm-safe; viewers resolve \r overwrites)

How it works

tlog is two decoupled halves that only meet at the filesystem: a write path that lives inside your training process, and a read path (the viewers) that runs anywhere that can see the same disk. There is no daemon, no database, no socket between them — a run is a directory:

runs/<project>/<name>__<timestamp>__<id>/
├── meta.json          # identity + environment snapshot + restart history
├── config.json        # your hyperparameters (vars(args))
├── metrics.jsonl      # one JSON object per log() call, append-only
├── system.jsonl       # sampled GPU/CPU/RAM
├── console.log        # teed stdout/stderr
├── launch.sh          # captured sbatch script (under SLURM)
├── diff.patch         # uncommitted git changes
└── media/             # PNGs + index.jsonl mapping them to (key, step)

The write path never blocks training

log() serializes one JSON line and appends it. Lines are written whole and flushed, so a crash loses at most the line in flight and can never corrupt history; fsync runs on a 30s timer to bound hard-failure data loss without paying sync cost per step. Everything slow happens off the hot path: git diff / nvidia-smi / scontrol captures run in a background thread after init, system sampling and the liveness heartbeat are daemon threads, and framework versions are read from sys.modules instead of importing anything.

Preemption-safe by construction

SLURM requeues a preempted job with the same job id and bumps SLURM_RESTART_COUNT. init(resume="auto") (the default) detects that, finds the run directory it created before the preemption, and keeps appending — recording a restart event in meta.json. Restarting from an older checkpoint re-logs some steps; instead of rewriting files (dangerous), readers keep the last value logged per (metric, step), so charts come out continuous and the storage stays strictly append-only. Explicit resume: tlog.init(id="a1b2c3", resume="must").

The read path is one engine with three faces

store.py discovers runs, tails JSONL incrementally (remembering byte offsets, parsing only complete new lines), applies keep-last dedup, and downsamples with min/max/mean buckets — a one-step loss spike survives being squeezed into a 200-px chart instead of being averaged away. Debiased EMA smoothing (same formula as wandb) sits on top. The three viewers are just renderers over this engine:

• TUI: each terminal cell is a 2×4 braille dot grid, so a tmux pane becomes a pixel canvas; charts are drawn with Bresenham lines and repainted on the alternate screen buffer. Pure ANSI — no curses, works over any SSH.
• Web: a stdlib ThreadingHTTPServer with JSON endpoints; the browser polls every 3s and refetches only runs whose files changed (mtime-keyed).
• Export: the same frontend with data, images (base64), and uPlot inlined into one HTML file. One codebase, a mode flag, two surfaces.

Liveness without IPC

A daemon thread touches heartbeat every 15s. Viewers call a run running if the heartbeat is fresh, finished if finish() marked it, and dead if neither — which is how a SIGKILLed job shows up correctly with no process ever being asked.

Distributed training

tlog.init() is a no-op on non-zero ranks (it checks the RANK env var set by torchrun/SLURM), so you can call it unguarded — or keep your existing if rank == 0: guard; both are fine.

Migrating from wandb

-import wandb
+import tlog

-wandb.init(project=args.project, name=args.name, config=vars(args))
+tlog.init(project=args.project, name=args.name, config=vars(args))

-wandb.log(avg, step=step)
+tlog.log(avg, step=step)

-wandb.finish()
+tlog.finish()

Runs land in ./runs by default; set TLOG_DIR=/scratch/$USER/runs (or pass dir=) to keep them on scratch.

The viewers in detail

tlog watch [runs...] — braille line charts with min/max bands, one page per metric group plus media and console pages; the grid auto-sizes to the pane and scrolls when a group has more charts than fit.

• Compare runs: tlog watch baseline high-lr (or name a project dir to take all its runs) overlays every metric wandb-style, one color per run, with a legend line. The r key cycles which run the console page shows.
• Media page: logged images render in the terminal — by default as half-block thumbnails (▀ + 24-bit color), which work in every terminal including through tmux over SSH. Runs are columns, steps are rows, exactly like the web media tab. On kitty/Ghostty (kitty graphics protocol) or iTerm2/WezTerm (inline images), true pixel images are used automatically — except inside tmux, which usually eats those escapes, so tmux gets half-block unless you force a protocol with --images kitty|iterm2. (--images off hides the page.)

Keys: ←/→ pages · ↑/↓ (or j/k) scroll charts / media steps / console history · m cycle media key · r cycle focused run · 1–9 force column count, 0 auto (or --cols N) · s smoothing (EMA 0 → 0.6 → 0.9 → 0.99) · l log scale · q quit.

tlog serve [root] — open http://localhost:8585 through VS Code Remote (auto port-forward) or ssh -L 8585:localhost:8585 cluster. Multi-run overlay charts with synced cursors, smoothing slider, log scale, a media tab laid out runs-as-columns × steps-as-rows for side-by-side recon/eval comparison, a config tab that highlights differing hyperparameters, and live console.

tlog export <runs...> -o report.html — the same UI frozen into a single file (images downscaled to ≤512px by default; --max-image-px 0 keeps originals). No server, no internet — works in VS Code's HTML preview.

tlog report spec.md [runs...] — custom pages: write plain markdown and drop in ```tlog blocks where you want live elements, then render to one self-contained HTML file (--open pops a browser). Prose narrates; blocks pull from the runs:

## Eval quality

FID is the one metric where high-lr finishes ahead.

```tlog chart
key: eval/fid
smooth: 0.9        # optional EMA; raw stays as a faint line
logy: true
runs: baseline, high-lr   # optional — defaults to the runs on the CLI
```

```tlog table
columns: config.lr, eval/fid min, eval/ssim max, loss/total last
```

```tlog images
key: eval/recon
last: 2            # or steps: 500, 1500
```

Three block types: chart (multi-run SVG overlay), table (one row per run; columns are metric keys with an optional min/max/last aggregator, or config.* values), and images (runs-as-columns × steps-as-rows grid). Because a run is just files and the spec is just markdown, reports are easy for both humans and coding agents to compose — ask an agent to inspect your runs and it can write the analysis and the page that shows the evidence (see examples/report.md).

Let a coding agent review your runs

Because a run is nothing but files on disk — metrics.jsonl, config.json, PNGs under media/ — a coding agent (Claude Code, etc.) sitting on the same cluster can inspect a run with no API key, no server, and no browser: it greps the metrics, opens the images, and reads the config directly. tlog report is the other half of that loop — it gives the agent a way to hand back what it found as something you can actually look at.

A typical remote-cluster workflow:

pip install "git+https://github.com/philippe-eecs/tlog"   # on the cluster
# ... training writes runs to ./runs as usual ...

Then, in a Claude Code session on that cluster:

"Compare baseline and high-lr. Look at the loss curves and the eval recon images, then write me a tlog report with your read on whether the higher LR helped."

The agent reads the JSONL and the PNGs, writes a spec.md with prose plus chart/table/images blocks, runs tlog report spec.md baseline high-lr, and you get a single self-contained spec.html — its analysis up top, the charts and side-by-side reconstructions as evidence below. scp it to your laptop, skim it, and reply with feedback; the agent revises the spec and re-renders. The diagrams, charts, and logs are all in one reviewable file, and the agent never needed anything but the run directory.

Demo without a GPU

python examples/fake_train.py --steps 2000 &
tlog watch

Prior art

trackio, aim, TensorBoard, and MLflow all live in adjacent space. tlog's niche is the combination: a zero-dependency stdlib-only core safe to drop into any training env, files you can grep as the source of truth, SLURM-native metadata + preemption semantics, a terminal dashboard designed for a tmux pane on a GPU cluster, and single-file HTML reports — in ~2,700 lines of Python you can read in an afternoon.

Tests

python -m pytest tests/

License

MIT