tinkerscope

A browser playground for Tinker-trained checkpoints. Point it at your training directory and it auto-discovers every run — no models.yaml, no registration — then chat with them, fan out N samples to see what a model usually says, branch like on claude.ai, and compare checkpoints side by side. The whole thing is also drivable live from your terminal (tinkerscope), so your AI agent can fire samples into the browser you're watching.

Quick start

export TINKER_API_KEY=...      # required to sample (discovery works without it)
uv tool install tinkerscope
tinkerscope ~/my-training-runs # scans the tree, auto-picks a port, prints the URL

Open the printed URL and you're in. tinkerscope DIR1 DIR2 … scans several trees at once. Set OPENROUTER_API_KEY too if you want OpenRouter reference models next to your checkpoints. Serving weights yourself? tinkerscope --vllm-url http://host:8000 (or TINKERSCOPE_VLLM_URL) lists every model a vLLM server serves in the picker (⚙) and samples it like a native model — token probabilities, the loom, the works (TINKERSCOPE_VLLM_API_KEY if the server has --api-key). Sharing one instance with a colleague? tinkerscope --multi-user gives every browser its own sidebar (models, params, open workspace) while the workspaces stay shared, and tinkerscope --session <id> drives one person's screen. Lost in the UI? The ? button in the sidebar explains every control and keyboard shortcut — or just ask your agent (guide skill below).

Agent skills

The repo doubles as a plugin marketplace shipping two skills: tinkerscope:guide makes your agent a help desk for the UI — ask it "how do I compare two checkpoints?" and it knows every button and shortcut — and tinkerscope:cli teaches it to drive the playground from a terminal (more below).

/plugin marketplace add Butanium/tinkerscope
/plugin install tinkerscope@tinkerscope

codex plugin marketplace add Butanium/tinkerscope
codex plugin add tinkerscope@tinkerscope

npx skills add Butanium/tinkerscope --agent <agent> -g -y

Features

Auto-discovery — no config files

tinkerscope scans your directories for checkpoints.jsonl + config.json (the two files every tinker_cookbook run drops) and surfaces one run per directory with its whole checkpoint trajectory. Runs Tinker can no longer sample are greyed out with the reason instead of failing on click. Every picker is type-to-filter with typo-tolerant fuzzy matching, and every model has a copy button for its tinker://… sampler path.

You can also add models straight from the UI: a raw Tinker base model, a loose checkpoint by sampler path, or any OpenRouter model to sit next to your checkpoints as a reference.

N-sample fan-out

Set n > 1 and one send draws N samples, rendered as cards — a quick read on what the model usually says. The usual knobs (temperature, max tokens, top-p), a thinking toggle (Off / On / Both — Both draws n with and n without, tagged), a system prompt, and an assistant prefill field.

Response distribution chart

The draws power a live chart. Define highlight rules (named colors + patterns) and each sample is bucketed by the rules it matches — "define a rule, see its prevalence per model" is one loop. A first token mode charts the model's own probability distribution over the first generated token next to what was actually sampled. Segments are clickable to inspect the exact samples underneath, and the chart live-updates while a batch streams.

Token probabilities

Native Tinker sampling stores per-token logprobs + top-5 alternatives on every turn. Flip the sidebar's Token probs toggle to paint a surprisal heat under the prose (or show the raw token stream), and hover any token for its probability and the alternatives the model weighed. Color by match tints each token by how much probability mass a highlight rule almost got — where did "red" nearly happen. And the alternatives aren't just for looking: click a token to loom — pick an alternative from its card and the model continues from exactly that point with the swap made (token-level replay, not a paraphrase), landing the counterfactual as a sibling branch you can cycle against the original, n samples at a time.

Parallel conversations with different models

Have the same conversation with several models at once: Add panel puts models side by side, and every message you send gets answered by each — same user turns, one response per model. Panels generate concurrently and each keeps its own branch tree.

Branch everything

Nothing is ever destroyed: regenerating, editing a turn, or drawing N samples creates sibling branches you can cycle through with ‹ k/N ›, and you can branch from the very start to keep several probe prompts in one workspace. Workspaces are named, persisted per project, and restored on restart. Every button has a tooltip, Shift/Ctrl unlock power variants, and the ? modal (or your agent, via the guide skill) covers the rest.

Search everything (Ctrl+K)

"I have a screenshot of a sample but no idea where I ran it": Ctrl+K searches every branch of every workspace — message text, thinking blocks, system prompts, workspace and model names — including samples that aren't the currently-visible sibling. Picking a match opens its workspace, flips the cyclers so the matched branch is the one on screen, and flashes the row. The same engine drives tinkerscope grep for agents.

Share packs

Bundle checkpoints + params + workspaces into one portable YAML. Models are addressed by public sampler paths, so a collaborator reproduces your exact setup with no local run dirs:

tinkerscope pack export demo.yaml     # author from your current setup
tinkerscope --pack <file|url>         # consume + serve

A pack is also a link: open ?w=<pack url> on a running instance and it installs (after asking) with no restart. Full doc: docs/PACK.md.

Publish a read-only site

Export the playground as a static site — no backend, no API key, hostable on GitHub Pages:

tinkerscope site export ./site --workspace "the good one"

Visitors browse the real thing: branches, threads, panels, the chart, token probabilities. Per-token logprobs are ~97% of the bytes — export a subset, or narrow them with --logprobs chart (keeps the turn each chart opens on) / last:N / none. A published site also works as a viewer for anyone's pack (?w=<pack url>, or drop the file on the page), and its read-only badge hands readers the command to run everything locally. Full doc: docs/STATIC_SITE.md.

Bring your agent — tinkerscope

The CLI hits the same API as the browser and shares its state, which makes the playground a place you and your AI agent work in together:

• It reads what you did. Your agent can walk every branch of your workspaces, pull full n-sample fan-outs, and grep across everything you ever sampled ("which workspace had that self-description prompt?").
• You watch what it does. Probes the agent fires land live as new threads in your open browser — its exploration stays auditable, thread by thread, instead of buried in a script's stdout.
• Its reads feed write-ups. Every read command takes --json, so a morning of browser sampling turns into an eval design or a report without re-running anything.

tinkerscope ls                                # List discovered training runs.
tinkerscope checkpoints <run>                 # List a run's checkpoints (name, step, whether it has a sampler).
tinkerscope open <run>                        # Select a run or model in single mode; the browser switches live.
tinkerscope chat <run> <prompt>               # Sample from a run's checkpoint; stream completions to stdout and the browser.
tinkerscope compare <run_a> <run_b> <prompt>  # Compare N runs on one prompt — A→primary, B→compare, --run extras→p-2,p-3,… all stream...
tinkerscope send [prompt]                     # Fire the prompt as a NEW THREAD at the CURRENT panels of the open workspace — the CLI twin of...
tinkerscope continue [prompt]                 # LOOM from an existing branch: rebuild the message history up to a target node and sample a...
tinkerscope battery <probes_dir>              # Fire a DIRECTORY of probe files as sequential `send`s — the reusable probe battery.
tinkerscope probe <run> [prompt]              # Sample ANY discovered model WITHOUT touching the browser or any workspace.
tinkerscope params                            # Show or SET the GLOBAL sampling params (system prompt, temperature, max tokens, n, thinking,...
tinkerscope url [selector]                    # Print the URL of the server this CLI is driving — the thing to hand the human when they ask...
tinkerscope sessions                          # List the SESSIONS a `--multi-user` server holds — one sidebar (panel selection, open...
tinkerscope state                             # Digest of what's on screen now: one block per panel, first/last-2 of the open workspace's...
tinkerscope threads                           # Cross-workspace index of every root THREAD — the find primitive for "where are my multi-turn...
tinkerscope ws [selector]                     # Browse saved WORKSPACES (multi-panel, branchable; `conv` is a back-compat alias).
tinkerscope samples [selector]                # Show every sibling response (the n-sample fan-out) at ONE fork, each with its CoT, plus a...
tinkerscope grep <pattern>                    # Search EVERY branch of saved workspaces — message content, thinking (`reasoning`) and thread...
tinkerscope node <node_id>                    # Locate a NODE ID anywhere in the saved workspaces and dump its record — the reverse index...
tinkerscope trash [action] [handle]           # Recover deleted branches.
tinkerscope refresh                           # Rescan the filesystem + re-probe sampling capabilities.
tinkerscope wait                              # Block until no generation is running — the sequential-wave primitive: fire, `tinkerscope...
tinkerscope serve [dirs...]                   # Serve the API + web UI for DIRS (bare `tinkerscope <dir>` is shorthand for this).
tinkerscope pack export <out>                 # Export the current setup to a pack YAML file.
tinkerscope site export <out>                 # Write a self-contained static site into a directory.

tinkpg, the CLI's old name, still works as a deprecated alias.

Param flags on a fire are per-call — they never clobber your browser sidebar. To point your agent at a turn you're looking at, the row's # button copies its <ws>:<panel>:<node> handle, which every --node takes. The tinkerscope:cli skill teaches your agent all of it, flags included (plugin/skills/cli/SKILL.md).

Tests

uv run pytest -q covers discovery and the API with zero remote calls. The pure frontend logic has ~20 bare-Node unit suites (npm test from web/), and scripts/smoke.sh runs the Playwright browser smokes against an isolated throwaway instance.

Development

./run.sh [DIR ...]          # dev mode: backend + vite dev server (hot reload)
./run.sh --build [DIR ...]  # packaged: build the web UI once, serve from one process

Installing as a tool while hacking? Use uv tool install -e . (editable) so the process runs your checkout. Backend changes need a process restart; web changes need npm run build (a pre-commit hook runs it on web/ commits) plus a browser refresh. When [project.scripts] entry points change, editable is not enough: the installed shims freeze their import target, so re-run uv tool install -e . to regenerate them (compat for the pre-unification tinkerscope shim is kept in serve.py:main).

for s in cli guide; do
  mkdir -p ~/.claude/skills/tinkerscope-$s
  ln -sf "$PWD/plugin/skills/$s/SKILL.md" ~/.claude/skills/tinkerscope-$s/SKILL.md
done

For orientation and contracts: CLAUDE.md (where everything lives), docs/API_CONTRACT.md (HTTP + SSE shapes), docs/BRANCHING_DESIGN.md (the branching model), docs/TODO.md (roadmap).

Credits

The UI is forked from Harry Mayne's tools/playground in HarryMayne/negation_neglect_working_repo. The core chat experience — streaming, n-sample fan-out, the response-distribution chart, the thinking toggle, the raw-text view, and the side-by-side compare — is his work. tinkerscope adds run auto-discovery, workspace branching, named/persisted workspaces, N-panel comparison, the token-probability views, share packs, the static-site export, the terminal-driving CLI, and standalone packaging on top.

tinkerscope's own code is MIT-licensed (see LICENSE). The upstream playground ships without a license; substantial portions of the UI and inference layer are Harry Mayne's work, retained here with attribution. If you build on this, keep that credit.