shidoshi

shidoshi is a local notebook for iterative work with LLMs and coding agents. You write Python, bash, notes and prompts in one notebook, and the notebook itself, in order, is the conversation: your code, its output and your notes are the context a prompt sees. Nothing runs or changes on its own. You run each cell, and you decide what the model gets to see.

Notebooks are ordinary .ipynb files, so anything you write in shidoshi opens in Jupyter, VS Code or GitHub.

Install and run

On macOS and Linux:

curl -LsSf https://raw.githubusercontent.com/tdchaitanya/shidoshi/main/install.sh | sh

On Windows (PowerShell):

powershell -ExecutionPolicy ByPass -c "irm https://raw.githubusercontent.com/tdchaitanya/shidoshi/main/install.ps1 | iex"

The script installs uv with Astral's installer if you don't have it, then runs uv tool install shidoshi and adds uv's tool directory to your PATH. It never uses sudo or your system Python. Run it again to upgrade. Set SHIDOSHI_VERSION=0.14 to install a specific version, or UV_NO_MODIFY_PATH=1 to leave your shell profile alone. install.sh is short if you'd like to read it first.

If you already have uv, or would rather install it yourself (winget install astral-sh.uv on Windows), install shidoshi as a uv tool directly. It gets its own isolated environment and a shidoshi command on your PATH, so one install works from any folder and doesn't touch your projects' environments:

uv tool install shidoshi    # later: uv tool upgrade shidoshi
uv tool update-shell        # if `shidoshi` isn't found in a new terminal

Then:

shidoshi                    # open the current directory
shidoshi ~/work/project     # open a folder
shidoshi analysis.ipynb     # open a notebook directly

shidoshi needs Python 3.13 or newer, and uv downloads it if your machine doesn't have it. uvx shidoshi runs it once without installing. pipx works too (pipx install --python python3.13 shidoshi), and so does pip install shidoshi into an environment you manage yourself.

shidoshi starts a local server, prints its URL and opens it in your browser. The URL carries a one-time access code, so use the full printed link when opening it in another browser. The server only listens on loopback addresses.

Useful flags: --port (default 8420, and the next free port if it is taken), --no-browser, and --ip (127.0.0.1, localhost or ::1). Run shidoshi --help for the rest.

The notebook

A notebook has four kinds of cell:

• Python cells run in a Jupyter kernel, in the Python environment you pick from the notebook's environment panel. Nothing starts until you run a cell.
• Bash cells run in a persistent shell in the notebook's folder.
• Notes are Markdown, with math, tables, checklists and images.
• Prompt cells are where you talk to a model.

Shift+Enter runs the current cell and selects the next; at the end it creates a blank Python cell ready for typing. Cmd/Ctrl+Enter runs in place. Notes render Markdown and prompts run Ask; neither shortcut runs adjacent cells. Python input() and getpass.getpass() accept replies inline, with password fields masked. Stop interrupts a pending input request.

A prompt cell can run in two ways. Ask sends one request: the cells above it become the conversation history, and the answer lands in the cell's output. The model can search the web, and you will see what it searched for. Agent hands the prompt to an agent that can act in the notebook. That is Codex or Claude Code running locally, if you have their CLIs installed, or the built-in agent. Codex and Claude Code show an approval card in the notebook for anything your permission settings don't already allow. The built-in agent can read your variables, run Python and bash, and propose new cells, and it runs code without asking first.

You control the context per cell. Pin a cell to keep it in every prompt's context. Exclude it to keep it out. Notes can be marked as asides, which prompts skip unless pinned.

If Ask or the built-in agent exceeds the model's context limit, it can retry with older unpinned cells omitted. A notice tells you what was omitted; the notebook stays intact. The built-in agent also summarizes older conversation memory. See context limits for pin protection, recovery limits, and how Claude Code and Codex differ.

Drop PNG, JPEG, GIF or WebP files between cells to add them as image notes. The files are saved next to the notebook and linked with ordinary relative Markdown, so keep them with the .ipynb when you move it. Ask and Agent send images from eligible cells above the prompt to models that accept images.

Python environments

Your notebook code should run in an environment of its own, not in the one shidoshi is installed in. With uv tool install, shidoshi's environment belongs to uv: packages you add to it can be removed on the next upgrade, and they can break shidoshi's own dependencies. The environment panel labels that kernel "Shidoshi's Python" and warns when a notebook uses it. In that kernel %pip, %uv, %conda, %mamba and %micromamba install nothing and point you to an environment of your own. Shell commands such as !pip can't be intercepted, so don't install with them there either.

The panel can create a separate environment for you, using uv, with no terminal needed. Each one is a small uv project under ~/.shidoshi/envs. %pip install in a notebook installs into that environment, and the Packages window adds, removes and upgrades packages with a preview first. You can also link a uv project you already have, or pick any Jupyter kernel registered on your machine.

Mark one environment "Use for new notebooks" and new notebooks start with it suggested. While a notebook is on Shidoshi's Python, the panel offers to create a default environment in one click. A notebook that already names a kernel keeps it, and nothing switches without you choosing it.

Models and keys

Ask uses OpenAI (through the Responses API), OpenRouter, or Anthropic (through the Messages API). The built-in agent supports OpenAI and OpenRouter; for Claude models there, use the Claude Agent harness. Choose the provider and model from the LLM provider panel in any notebook; each notebook remembers its own choice.

Keys come from OPENAI_API_KEY / OPENROUTER_API_KEY / ANTHROPIC_API_KEY if they are set. Otherwise enter one in the panel and keep it for the session, or save it to your OS keychain. Keys are never written to notebooks or browser storage. OPENAI_BASE_URL points the OpenAI provider at a proxy or compatible endpoint, which must implement /responses.

Codex and Claude Code use their own CLI login and settings.

web-ui/README.md covers kernels, credentials, the Debug toggle, reconnecting and branching in detail.

Coming from the Jupyter magics

Earlier versions of shidoshi were a set of Jupyter magics (%ask, %%agent, %%pin, %shidoshi config) with a TOML config file. Those are gone. The web notebook replaces them, and it can open your existing notebooks.

• %load_ext shidoshi no longer loads anything. Old %ask cells open as plain code cells. Their saved answers stay in place, but they are no longer prompts.
• If you ran shidoshi install-kernel, remove that kernel with jupyter kernelspec remove shidoshi.
• config.toml files are ignored. Choose the provider and model in the web UI.
• shidoshi web is now just shidoshi.

Development

To launch from a source checkout with a fresh UI, use:

./scripts/shidoshi.sh
./scripts/shidoshi.sh /path/to/notebook.ipynb --port 8422

The launcher requires Node/npm and uv, installs frontend dependencies from the lockfile, rebuilds the UI, and then runs uv run shidoshi from this checkout. It forwards all Shidoshi arguments and preserves the directory you launched from. A failed install or build stops the launch. It builds the code currently checked out; run git pull --ff-only yourself when you want to update it.

For separate setup and testing:

uv sync
npm --prefix web-ui ci && npm --prefix web-ui run build   # the UI the server serves
uv run pytest tests/unit tests/btp tests/test_import_rule.py

In a source checkout, Shidoshi's Python refuses %pip too, as it does for users. To install into the checkout's .venv from a notebook anyway, start the server with SHIDOSHI_ALLOW_INTERNAL_PIP=1 uv run shidoshi. Only the value 1 turns the guard off, and it applies to kernels started after that.

Integration tests in tests/integration/ call real models. They need OPENAI_API_KEY / OPENROUTER_API_KEY (or OPENAI_BASE_URL for a proxy):

uv run pytest tests/integration/ -v -m integration

To audit prompts, capture the exact requests sent to providers. Capture is off unless you pass the flag:

• shidoshi --capture-requests DIR routes Ask, the built-in agent, Codex and Claude Code through a local logging proxy. It records every provider request as sent, including the harness-owned prompts and tools Debug can't show.
• Files go to DIR/<session>/<notebook>/<run>/, one JSON file per request, with a run.json per run and a DIR/index.jsonl manifest. Responses are recorded as metadata only (status, usage, timing). Add --capture-responses to keep full response bodies too.
• Auth headers are stripped, bodies are redacted, and files are owner-only (0700/0600). A DIR inside the workspace is refused unless you pass --capture-in-workspace, and nothing is written into .ipynb.

See docs/request-capture.md for what each path captures and the known gaps.

See web-ui/README.md for running the frontend against a live-reloading dev server and for the browser test suites.

For PyPI releases, scripts/release.sh and the publish workflow rebuild the frontend and run scripts/verify_frontend.py. Both the wheel and source archive must contain exactly the fresh frontend before publishing can proceed.

The test workflow uses Linux x64 runners with the shidoshi-ci label on luge-per-middlegame.exe.xyz: gondola-tarte-1 and gondola-tarte-2. Each runs as a separate non-sudo user with its own workspace and caches under /opt/shidoshi-runners/runner-1 or runner-2. Both systemd services start on boot; inspect them on the VM with sudo systemctl status 'actions.runner.*'. Chromium system dependencies are installed on the VM; rerun Playwright's install-deps chromium as an administrator when an upgrade requires new ones. Adding runners to this same VM also requires assigning distinct browser test ports in .github/workflows/test.yml.

Two additional runners, neon-1 and neon-2, run in separate unprivileged Docker containers on the Ubuntu homelab. Each has a two-CPU / 4 GiB RAM limit, separate workspace/cache volumes and its own network namespace, so browser ports can overlap safely. Jobs cannot access homelab files or its Docker socket. Their setup, tests and operations live in homelab/services/github-runners. Run the manual runner-smoke workflow to validate both exact runner labels through dependency installation, builds, package verification and browser tests. These older runners increase capacity but are slower per job; the test workflow allows 90 minutes for either job.

CI stores pytest data in $RUNNER_TEMP/pytest, which the runner clears between jobs, and prunes uv's dependency cache after each suite, including failed runs. Before installing dependencies, CI checks for at least 4 GiB and 500,000 free inodes on the temporary filesystem, leaving headroom on each runner host. The old retained pytest runs consumed 6.8 GiB and over 1.1 million inodes across the VM. If the check fails, inspect df -h, df -i, and retained test data before retrying.

License

Apache License 2.0. See LICENSE.