shidoshi

An opinionated way to augment Jupyter Lab for iterative work.

shidoshi adds %ask / %%ask magics to Jupyter that let you talk to an LLM from inside a notebook — using the notebook itself, in order, as the conversation history. No separate chat pane, no copy-pasting context: your code cells, their outputs, and your notes are the context.

Install

Requires Python ≥3.13 and JupyterLab. Set an API key before use:

export OPENAI_API_KEY=sk-...       # for the openai provider (default)
export OPENAI_BASE_URL=...         # optional, e.g. to point at a proxy
export OPENROUTER_API_KEY=...      # for the openrouter provider

Installing across Jupyter environments

%load_ext shidoshi runs import shidoshi inside the running kernel process. That means shidoshi has to be installed into whichever Python environment the kernel you're using actually runs in. It ships a small shidoshi command for setup, but the library itself is not a standalone tool — so uvx / uv tool install (which run a tool in an isolated subprocess, separate from any kernel) don't apply here.

• Per-project venv with its own JupyterLab (e.g. a uv-managed project): add shidoshi as a normal dependency of that project.

  uv add shidoshi
  # or: pip install shidoshi
  

• One shared JupyterLab, many kernels (each notebook's kernel points at a different project venv registered via ipykernel install): install shidoshi into each kernel's venv. Installing it only where JupyterLab itself lives will not make it importable from other kernels.

  # inside the venv backing a given kernel
  uv add shidoshi
  # or: pip install shidoshi
  

Skipping %load_ext — the shidoshi kernel

To avoid typing %load_ext shidoshi in every notebook, register a kernel that loads it for you:

shidoshi install-kernel

Pick Python 3 (shidoshi) from the Jupyter kernel list and the magics are already there. Nothing else changes: it is a stock Python kernel running this environment's interpreter — your imports, variables, and debugger all work exactly as before. The generated kernel.json just appends --IPKernelApp.extensions=shidoshi to the normal ipykernel_launcher command, with an absolute path to this environment's Python.

With no location flag, this installs into your per-user kernel directory — the only location every Jupyter on the machine can find, regardless of which environment actually launches it. After installing, the command runs jupyter --paths for you and prints a warning if the kernel it just wrote won't actually be visible to the jupyter on your PATH — worth reading if you pass --sys-prefix or --prefix and the kernel doesn't show up in the launcher.

Useful flags:

Installing is refused if a kernelspec of that name already exists, so it won't quietly replace one you made by hand. Register one per environment with a distinct --name.

Remove it with jupyter kernelspec remove shidoshi.

With uv

Add shidoshi to the project, then register the kernel from inside it. Which location flag you need depends on where JupyterLab itself runs from, because Jupyter only searches its own sys.prefix, your user directory, and the system directory:

uv add shidoshi

# A: JupyterLab in an ephemeral env (uv's default suggestion).
#    Its sys.prefix is a uv cache dir, so --sys-prefix would be invisible.
uv run shidoshi install-kernel --user
uv run --with jupyter jupyter lab

# B: JupyterLab as a project dependency — sys.prefix *is* the project venv.
uv add --dev jupyterlab
uv run shidoshi install-kernel --sys-prefix
uv run jupyter lab

B keeps the kernel scoped to the project and disappears with the venv; A is the one that works with uv run --with jupyter — and matches install-kernel's default, so uv run shidoshi install-kernel (no flag) does the same thing. If a freshly installed kernel doesn't show up in the launcher, run jupyter kernelspec list the same way you start Lab — that prints exactly the directories being searched, or trust install-kernel's own post-install warning, which runs that same check for you.

Either way this replaces the kernel step in uv's Jupyter guide — you don't need uv run ipython kernel install --env VIRTUAL_ENV ... as well, because install-kernel records VIRTUAL_ENV for you when it detects a venv.

That variable matters more than it looks. uv pip install resolves its target from VIRTUAL_ENV (falling back to CONDA_PREFIX, then a base interpreter) — never from the kernel that's running. So if you start Jupyter from a conda-activated shell, a kernel without VIRTUAL_ENV will import from your project venv while !uv pip install quietly installs into your conda base. Pinning it keeps both views on the same environment.

Two related notes:

• !uv add was always safe — it finds the project by walking up for pyproject.toml, so it ignores VIRTUAL_ENV and targets the project venv either way.
• %pip install needs uv venv --seed; uv venvs have no pip in them by default. Prefer !uv add.

Pass --env to set anything else the kernel should launch with (repeatable), including an override for VIRTUAL_ENV:

uv run shidoshi install-kernel --sys-prefix --env OPENAI_BASE_URL=http://127.0.0.1:18080/v1

Because the spec pins an absolute interpreter path, it can only ever start the environment shidoshi is installed in. That is the advantage over the ipython_config.py route below: ~/.ipython is shared by every Python environment under your $HOME, so putting the extension there makes every kernel on the machine try to import shidoshi, including ones that don't have it.

c.InteractiveShellApp.extensions = ["shidoshi"]

Quickstart

%load_ext shidoshi

(skip this line if you're on the Python 3 (shidoshi) kernel)

%%ask
What does the `history.build_history` function in this file do?

The response streams into the cell's output as Markdown.

Magics reference

• %ask <prompt> — line magic for a one-line prompt.
  • Prefix with model| or provider:model| to override the configured default model for just this call, e.g. %ask openrouter:openai/gpt-4o|summarize this.
  • Add --debug anywhere on the line to also show the full request/response payload.
• %%ask [model] — cell magic; the whole cell body is the prompt (multi-line is fine, and it can reference images via Markdown ![]()/<img> syntax or bare local file paths — they're inlined as base64). An optional model name on the magic line overrides the default for this call. Also supports --debug.
• %%skip — runs the cell normally, but the cell is left out of the context sent to the model entirely. Use it for scratch or exploratory cells you don't want the model to see.
• %%pin — runs the cell normally; its content and output are always included in context and are exempt from the auto-trim behavior below. Use it to protect a fact, constant, or definition you don't want dropped over a long session.
• %%agent [model] — a multi-step, tool-using agent instead of a one-shot answer. See below.
• %agent_resume approve|reject|edit — answers a %%agent turn that paused for approval (only relevant if agent.approvals is configured — see Configuration).
• %shidoshi config ... — inspect and edit shidoshi's own config from inside a notebook cell. See Configuration.

%%agent — the agentic magic

Where %%ask answers once from what the notebook shows, %%agent can take several steps and look at what the kernel actually holds — including variables no cell output ever displayed. It runs deepagents in-process, as a backend parallel to %%ask's; %%ask is unchanged.

%%agent
Which of my dataframes has missing values, and where?

The answer renders as Markdown with the agent's steps in a collapsed 🧠 Agent steps panel above it.

Tool rungs

--tools chooses what the agent is allowed to do. The default is actor: model-written code runs in your live namespace, with no confirmation step. That is the reason %%agent exists rather than being a slower %%ask, so it does not sit behind a flag — but know that it is what a bare %%agent cell does.

--no-tools is the way back down — short for --tools none, for when you want the conversation without the hands. --tools proposer is the middle ground: the agent hands you a cell and your Run button is the approval step, with no separate permission prompt to click.

Set a different default in config with agent.permission_rung — permission_rung = "proposer" restores the gated behavior for every cell. Go a step further and require an explicit approval before run_code itself even runs — see Configuration.

Memory

Each notebook gets one conversation thread, seeded once from the cells above the first %%agent call. Later cells continue that conversation rather than rebuilding context each time.

The thread lasts as long as the kernel and no longer. This is deliberate: what the agent remembers is largely kernel state — variables it listed, values it read — and a restart destroys exactly that. A conversation that outlived the kernel would keep tool results reporting variables that no longer exist, worded as fact. So a restart clears the thread and the notebook is read again as it currently stands, which is what rerunning it from the top means anyway.

Use --fresh to opt out of the thread entirely and get %%ask-style behavior — context rebuilt from cells, nothing remembered — or --thread NAME to keep a side conversation separate.

Inspecting a turn

--debug works as it does for %%ask, adapted to a graph that runs more than once. Above the answer you get the request panel — provider and model, the composed system prompt, every message the model will see, and the tools this rung binds — followed by a live panel of raw LangGraph events as the turn runs. Messages already in the thread are labelled from thread, because on a continued conversation only your new prompt is passed in; the rest comes from the checkpoint.

The stream panel is LangGraph's fullest per-step view — task, task_result and checkpoint events with step numbers, the middleware nodes that a plain update stream never names, and a per-task error when one fails. Failed steps are flagged in the summary line so you don't have to expand them to find the one that broke. A running token count sits above it, for both providers.

Token deltas are collapsed into a per-node count, so what you read is the node transitions — model → tools → model — rather than several hundred one-token lines.

An unrecognised flag is an error rather than a no-op, so a typo'd --tools tells you instead of quietly running with the default.

Getting help

%ask --help and %agent --help print the flags, and for %%agent the rung table. Use the single-% line form: IPython rejects a cell magic whose body is empty before the magic itself runs, so %%agent --help on its own can never reach us.

%agent [model |] your request is also a one-line shorthand for %%agent, matching %ask. Flags are cell-form only — on one line a bare token cannot be told apart from the first word of a question.

How context is built

Every prior cell in the notebook — up to the one you're currently running, and accounting for kernel restarts — is turned into conversation history automatically:

• Markdown cells become background text/image context (treated as notes or reference material, not instructions).
• Regular code cells appear as fenced code plus their text/image outputs.
• Prior %ask / %%ask cells become real user/assistant turns. Their responses are reused from a per-cell cache rather than re-sent, so replaying history doesn't resend answers the model already produced.
• %%skip cells are dropped entirely.
• %%pin cells are always kept.

Automatic context-length handling

If a request is rejected for exceeding the model's context window, shidoshi automatically retries, dropping the oldest trimmable history units first (markdown cells, then plain code cells, then whole ask+response pairs — %%pin cells are never dropped), up to 20 times. A banner reports how many cells were dropped so you know context shrank.

Providers & tools

• openai (default) — uses the OpenAI Responses API.
• openrouter — uses OpenRouter's chat-completions API; select it with the provider:model prefix, e.g. openrouter:anthropic/claude-3.5-sonnet.

%ask/%%ask requests have the built-in web_search tool attached by default, so the model can search the web when it needs current information; web_fetch also exists and can be turned on via config (tools.web_fetch.enabled = true), off by default. Both are configurable per notebook/project — see Configuration. %%agent does not have web tools yet, regardless of config — that's still on the roadmap.

Debug mode

Add --debug to %ask/%%ask to render a collapsible, syntax-highlighted panel showing exactly what was sent (system prompt, full message history, tools) and every raw event streamed back — useful when the model's behavior is surprising and you want to see the actual payload.

Configuration

shidoshi is configured through layered TOML files — global, project, and per-notebook — merged in that order, with a per-invocation override always winning. Nothing lower is ever silently overridden without a trace: shidoshi config explain <key> tells you exactly which file set the value you're seeing.

Where config lives

A project or notebook config isn't found by convention alone — write one:

shidoshi config init --project            # a short starter file
shidoshi config init --project --full     # every available setting, with defaults filled in

(or %shidoshi config init --project from a notebook cell — every shidoshi config ... command below also works as %shidoshi config ....)

The essentials

config_version = 1

[defaults]
profile = "default"

[profiles.default]
model = "openai:gpt-5.5"

[profiles.default.generation]
reasoning = { effort = "low" }

[agent]
permission_rung = "actor"   # none | observer | proposer | actor

The old flat keys (default_model, agent_model, agent_tools, reasoning_effort, ask_color, skip_color) still work — shidoshi migrates them automatically — but the nested form above is what shidoshi config init writes now, and it's the only shape that can express everything else on this page (profiles, tool policy, context policy, approvals).

Inspecting and editing config

shidoshi config path                # which files are actually in effect, in precedence order
shidoshi config show                # the fully merged, effective config
shidoshi config show --sources      # ...annotated with which file set each value
shidoshi config explain agent.permission_rung   # one setting's value and where it came from
shidoshi config validate            # parse/schema-check every layer, and flag pasted-in secrets
shidoshi config doctor              # validate, plus orphaned sidecars and missing credential env vars
shidoshi config doctor --online     # opt in to current provider metadata checks
shidoshi config set agent.permission_rung observer --project
shidoshi config unset agent.permission_rung --project
shidoshi config schema              # the complete field-by-field surface, as JSON Schema
shidoshi config schema --html       # ...or a single offline, browsable reference page

Each of these also runs as %shidoshi config ... inside a notebook cell, with two differences: edit prints the resolved path instead of spawning $EDITOR (a blocking subprocess doesn't belong in a kernel cell), and a bare call with no --global/--project/--notebook flag automatically includes the current notebook's own sidecar.

Kernel-local overrides are explicit and inspectable:

%shidoshi config session set generation.temperature 0.2
%shidoshi config session show
%shidoshi config session unset generation.temperature
%shidoshi config session clear

For one call only, repeat --set key=value on %ask/%%ask/%%agent, or use --profile NAME. The whole override is parsed and validated before a request, checkpoint, or pending approval is touched.

Provider policy is translated separately for raw OpenAI Responses, ChatOpenAI, raw OpenRouter, and ChatOpenRouter. In particular, OpenRouter's default data_collection = "deny" and require_parameters = true are sent on the request; custom OpenAI-compatible endpoints can choose providers.openai.api_mode and instruction_role explicitly. Configuration display and diagnostics recursively redact credentials, authorization headers, secret-like metadata, and URL query values.

For newly released provider request parameters that Shidoshi does not yet name, use the provider-scoped kwargs-style escape hatch:

[providers.openai.extra_body]
future_option = true

[providers.openrouter.extra_body]
future_option = true

The same table can be set under a profile's providers section. These values reach both the raw and LangChain-backed request paths. Shidoshi rejects keys that collide with request fields it owns (model, tools, reasoning, storage, routing/privacy policy, and limits), recursively redacts the passthrough in diagnostics, and includes it in model-build fingerprints. extra_body is for JSON request-body parameters; arbitrary Python constructor objects remain Python extension points because TOML cannot represent or validate them safely.

What's configurable

Model and generation settings (per named profile — select one with defaults.profile, or ask.profile/agent.profile to use a different one for each magic), provider transport, which tools %ask/%%agent may use (tools.web_search, tools.web_fetch, tools.create_cell, kernel-tool output limits and secret redaction), notebook context policy (what gets included, image handling, trim limits), local response caching, and %%agent's permission model — including, if you want it, a real human-in-the-loop approval gate:

context.include_markdown, include_code, include_outputs, and include_images control notebook history. The nested image policy (allow_local_files, allow_remote_urls, and max_file_bytes) applies to images in both the current prompt and notebook history.

[agent.approvals.run_code]
enabled = true
allowed_decisions = ["approve", "edit", "reject"]

With this set, an actor-rung run_code call pauses instead of running immediately. Answer it with %agent_resume approve, %agent_resume reject [reason], or %agent_resume edit <json args>.

shidoshi config schema --html is the complete reference — every setting, its type, and its default, generated straight from the schema shidoshi itself validates against, so it can never drift out of date. Fields that cannot yet be enforced safely (for example durable agent persistence or host-backed skill/memory paths) fail validation with a precise reason; shidoshi does not accept them as inert promises.

Development

uv sync
uv run pytest tests/unit tests/btp -v

Integration tests under tests/integration/ require a live OPENAI_API_KEY (or a proxy via OPENAI_BASE_URL) and are run with:

uv run pytest tests/integration/ -v -m integration

License

Apache License 2.0 — see LICENSE.