napari-chat-assistant 1.4.3

Local chat assistant scaffold for napari image-analysis workflows

napari-chat-assistant

---

Local Ollama-powered assistant for napari image-analysis workflows.

napari-chat-assistant adds a dock widget inside napari that understands the active viewer session, runs built-in image-analysis actions, and generates executable napari Python code when a request goes beyond the current toolset.

It is designed for local interactive work, repeatable workflows, and gradual automation rather than cloud chat or fully opaque “one-click AI”.

The current direction is a deterministic, layer-aware assistant: the plugin profiles loaded napari layers first, then uses that structured context to guide tool choice and generated code.

What's New In 1.4.3

• added built-in image grid view for side-by-side comparison of open image layers
• added a built-in action to turn image grid view off and restore hidden non-image layers
• added built-in plugin UI help so users can ask what controls do and get short usage tips
• expanded help coverage for Library, Prompt, code actions, model controls, telemetry, diagnostics, status, and prompt tips
• kept experimental SAM2 under Advanced rather than the default toolbar
• kept the clearer Load, Unload, Test, Setup model flow and improved status wording

For older release history, see CHANGELOG.md.

Overview

Current capabilities include:

• connect to a local Ollama server
• discover and unload local models from the plugin UI
• inspect the selected layer or named layers with structured layer summaries
• profile loaded layers with deterministic semantic and workflow-aware metadata
• run built-in, registry-backed image-analysis tools from natural-language requests
• support common microscopy workflows such as enhancement, thresholding, cleanup, measurement, projection, and cropping
• support ROI-aware inspection and grayscale value extraction from Labels and Shapes layers
• automate batch actions across multiple layers when the request applies to all compatible inputs
• generate napari Python code only when no built-in tool is a better fit
• review, copy, validate, and run assistant-generated code from the plugin UI
• paste and run your own Python directly from the Prompt box with Run My Code, without opening QtConsole
• reuse built-in demo packs for EM-style grayscale data, fluorescent RGB cell data, SNR sweeps, and messy mask cleanup tests
• save, pin, tag, rename, and reuse prompts through the local Library
• save, tag, rename, and run reusable code through the Library Code tab
• keep built-in examples alongside recent and saved items in the Library
• delete selected built-in, recent, saved, or code items from the Library and clear unpinned recent history while keeping saved and pinned items
• keep bounded session memory from approved prior turns
• reject the last assistant outcome from session memory with a thumbs-down control
• optionally configure and run experimental SAM2 segmentation from Advanced
• optionally open ND2 conversion, spectral viewer, and spectral analysis widgets from napari-nd2-spectral-ome-zarr

The current default model is:

• nemotron-cascade-2:30b

Why This Plugin

Most chat interfaces are detached from the actual napari session. This plugin keeps the assistant inside the viewer and grounds its behavior in the live state of the current napari workspace:

• loaded layers
• the selected layer
• layer shape, dtype, and transforms
• deterministic semantic layer profiling
• labels and mask statistics
• built-in local tool execution
• local Python code generation when needed
• bounded session memory

The goal is not a generic chatbot inside napari. The goal is a layer-aware assistant that can grow into a practical microscopy workflow workbench.

Local-first by design

The assistant runs on local open-weight models through Ollama:

• no API key required
• no internet dependency
• no cloud services
• no data leaves your workstation

This makes it a better fit for research workflows where users want interactive help, repeatable analysis steps, reusable demo and prompt packs, and direct control over both data and models.

Current Features

Session-aware tools

The assistant currently supports built-in tools for:

• listing all layers
• inspecting the selected layer
• inspecting a specific named layer
• CLAHE contrast enhancement for grayscale 2D and 3D images
• batch CLAHE across multiple image layers
• Gaussian denoising for grayscale image layers
• threshold preview
• threshold apply
• batch threshold preview and apply
• mask measurement
• batch mask measurement
• mask cleanup operations such as hole filling, small-object removal, and largest-component selection
• connected-component labeling for binary masks
• per-object measurement table summaries for labels layers
• max-intensity projection for 3D grayscale images
• cropping one layer to the bounding box of another layer
• ROI inspection and grayscale value extraction from labels or shapes regions
• registry-backed tool execution as the foundation for future workflow and pipeline expansion

Layer inspection is now backed by a deterministic profile object that includes:

• semantic_type
• confidence
• axes_detected
• source_kind
• metadata flags such as multiscale, lazy/chunked, channel metadata, and wavelength metadata
• recommended and discouraged operation classes
• evidence buckets for debugging and future adapter work

Supported mask operations:

• dilate
• erode
• open
• close
• fill_holes
• remove_small
• keep_largest

Additional built-in workflow tools currently exposed through chat include:

• gaussian_denoise
• remove_small_objects
• fill_mask_holes
• keep_largest_component
• label_connected_components
• measure_labels_table
• project_max_intensity
• crop_to_layer_bbox
• inspect_roi_context
• extract_roi_values

Code generation workflows

When a request is not covered by a built-in tool, the assistant can return napari Python code instead of guessing or forcing the wrong tool.

Generated code can be:

• copied to the clipboard
• reviewed and executed from the plugin

You can also paste your own Python directly into the Prompt box and run it from the plugin with Run My Code, without switching to QtConsole.

Use assistant-generated code when you want a reusable script or need custom logic beyond the current built-in tools.

Use Run My Code when you already have Python you want to test quickly inside the current napari session.

Optional ND2 and spectral integration

If napari-nd2-spectral-ome-zarr is installed, the assistant can open:

• the ND2-to-OME-Zarr export widget
• the Spectral Viewer widget
• the Spectral Analysis widget

This lets chat act as an entry point for Nikon ND2 conversion and spectral workflows without rebuilding those UIs inside this plugin.

Install links:

• GitHub: https://github.com/wulinteousa2-hash/napari-nd2-spectral-ome-zarr
• napari Hub: https://napari-hub.org/plugins/napari-nd2-spectral-ome-zarr.html

Experimental SAM2 integration

Version 1.4.3 includes an experimental SAM2 path for users who want box-prompt or point-prompt segmentation inside napari without making SAM2 part of the default assistant workflow.

Behavior:

• SAM2 is accessed from Advanced, not from the main toolbar
• SAM2 Setup is always available from Advanced
• SAM2 Live stays disabled until the backend is configured and passes readiness checks
• the rest of the assistant remains usable even if SAM2 is not configured

Current setup expects:

• a working Python environment that already includes the dependencies required by your SAM2 wrapper
• an external SAM2 project path
• a wrapper module exposed as sam2_wrapper.py or sam2_wrapper/__init__.py
• segment_image_from_box(...) for box workflows
• segment_image_from_points(...) for point workflows if you want live point prompting
• a valid checkpoint path
• a valid config path

Typical setup flow:

1. Start napari from the environment that contains your SAM2 dependencies.
2. Open Plugins -> Chat Assistant.
3. Open Advanced -> SAM2 Setup.
4. Enter the SAM2 project path, checkpoint path, config path, and device.
5. Click Test.
6. Save the settings.
7. Open Advanced -> SAM2 Live when the backend reports ready.

Selective Session Memory

The assistant now includes bounded session memory with three states:

• provisional
• approved
• rejected

Behavior:

• new assistant outcomes start as provisional
• successful follow-up actions can promote them to approved
• only approved items are sent back to the model as session_memory
• current viewer context and current layer profiles always override memory
• Thumbs Down Last Answer rejects the most recent memory candidate for the current session

This is intentionally not full transcript memory. The model is still grounded primarily in the current napari viewer state.

Library

The assistant includes a persistent Library for repeatable workflows and reusable code:

• built-in starter prompts
• built-in demo packs and reusable code examples in the Code tab
• recent prompts captured automatically
• saved prompts for reusable tasks
• pinned prompts for high-frequency workflows
• recent and saved code snippets in a separate Code tab

Interaction:

• single click loads a prompt or code snippet into the editor
• double click sends a prompt directly or runs a code snippet
• right click can rename or edit tags for saved and recent prompt/code items
• multi-select supports Shift/Ctrl selection for batch actions
• Delete can remove saved prompts, recent prompts, code snippets, or hide built-in prompts
• Clear removes unpinned recent prompt/code items while keeping saved and pinned items

Logic:

• saved means a user-managed prompt you want to keep as your own reusable entry
• pinned means keep this prompt surfaced at the top of the library
• a prompt can be pinned without being saved
• built-in prompts are shipped examples; deleting them hides them from the current local library view
• built-in code entries include demo packs and starter Run My Code examples
• built-in code entries remain visible even if the same snippet also appears in recent history
• code snippets can be tagged and renamed so they are easier to reference later in workflows

This is designed for users who want repeatable automation without committing everything to full scripting.

Requirements

• Python 3.9+
• napari
• Ollama installed locally and running on the same machine
• at least one local Ollama model such as nemotron-cascade-2:30b

Core Python dependencies used by the plugin are installed with the package itself.

Optional:

• napari-nd2-spectral-ome-zarr for ND2 export, spectral viewer, and spectral analysis integration
• external SAM2 project, weights, and config if you want the experimental SAM2 workflow

Notes:

• The plugin does not bundle the Ollama server or model weights.
• Model memory requirements vary substantially by model tag.
• Larger local models may require significant RAM or VRAM.

Tested during development on an NVIDIA DGX Spark workstation.

Installation

1. Install Ollama

Install Ollama on the same machine as napari, then start the local server:

ollama serve

Pull at least one model before using the plugin:

ollama pull nemotron-cascade-2:30b

Optional alternatives:

ollama pull qwen3-coder-next:latest
ollama pull qwen3.5
ollama pull qwen2.5:7b

2. Install the plugin

For normal use:

pip install napari-chat-assistant

For development:

git clone https://github.com/wulinteousa2-hash/napari-chat-assistant.git
cd napari-chat-assistant
pip install -e .

Release

This package is published to PyPI so napari Hub can discover it.

For maintainer release instructions and PyPI publishing setup, see RELEASING.md

Usage

1. Start napari.
2. Open Plugins -> Chat Assistant.
3. Leave Base URL as http://127.0.0.1:11434 unless your Ollama server is elsewhere.
4. Choose a model from the Model dropdown or type a model tag manually.
5. Click Test.
6. Start chatting, or use the Library for repeatable tasks and reusable code.

If you already have Python code you want to try, paste it into the Prompt box and click Run My Code. This runs viewer-bound code directly inside napari without opening QtConsole.

The assistant works best when prompts describe a concrete action. Natural language is fine.

Examples:

• Inspect the selected layer
• Preview threshold on em_2d_snr_mid
• Apply gaussian denoise to em_2d_snr_low with sigma 1.2
• Fill holes in mask_messy_2d
• Remove small objects from mask_messy_2d with min_size 64
• Keep only the largest connected component in mask_messy_2d
• Measure labels table for rgb_cells_2d_labels
• Create a max intensity projection from em_3d_snr_mid along axis 0
• Crop em_2d_snr_high to the bounding box of em_2d_mask with padding 8
• Inspect the current ROI
• Extract ROI values from em_2d_snr_mid using em_2d_mask

Demo Packs

Use the Library Code tab to load built-in demo packs for repeatable testing.

Current demo packs include:

• EM 2D SNR sweep
• EM 3D SNR sweep
• RGB cells 2D SNR sweep
• RGB cells 3D SNR sweep
• messy masks 2D/3D

These create named layers so you can test built-in tools quickly without hunting for sample data. Labels layers from the demo packs can also be used as ROIs for ROI inspection and value extraction.

Example pipeline:

1. Run the EM 2D SNR Sweep demo pack.
2. Apply gaussian denoise to em_2d_snr_low with sigma 1.0
3. Preview threshold on em_2d_snr_low_gaussian
4. Apply threshold now on em_2d_snr_low_gaussian
5. Fill holes in em_2d_snr_low_gaussian_labels
6. Remove small objects from em_2d_snr_low_gaussian_labels_filled with min_size 64
7. Keep only the largest connected component in em_2d_snr_low_gaussian_labels_filled_clean
8. Measure mask on em_2d_snr_low_gaussian_labels_filled_clean_largest

UI Overview

Model Connection

• local Ollama base URL
• model picker with discovered local models
• Test
• Use
• Setup help with install, ollama serve, and model pull examples
• Unload

Library

• built-in prompts
• built-in demo packs and starter code in the Code tab
• recent prompts
• saved prompts
• recent and saved code snippets in the Code tab
• pinned prompts
• saved keeps your own reusable copy
• pinned keeps a prompt at the top regardless of whether it is built-in, recent, or saved
• single click to load
• double click to send from Prompts or run from Code
• right click to rename or edit tags for saved/recent prompt and code items
• Shift/Ctrl multi-select for batch actions
• Delete works on built-in, recent, saved, and code items
• Clear keeps saved and pinned items and clears unpinned recent items
• A- and A+ adjust library font size in small steps

Chat

• multi-line prompt box
• Enter to send
• Shift/Ctrl/Alt+Enter for newline
• transcript showing user messages, assistant replies, tool results, and generated code

Code Actions

• Reject
• Run Code
• Run My Code
• Copy Code
• Advanced
• Help

Run Code is for assistant-generated code that has been staged in the chat.

Run My Code is for your own pasted Python from the Prompt box when you want to test or iterate directly inside napari without opening QtConsole.

Advanced contains optional integrations such as experimental SAM2 setup and live preview.

Current Context

• current layer summary from the active napari viewer
• shortened layer names and a compact per-layer summary to avoid over-stretching the left column

Session

• Activity tab shows local status updates, model connection messages, tool execution messages, and code execution/copy actions
• Telemetry tab contains the optional telemetry controls
• Diagnostics tab provides access to the app log and crash log
• color-highlighted path entries for assistant log, crash log, telemetry log, prompt library, and session memory
• Enable Telemetry switch for advanced users
• Summary, Log, and Reset only when telemetry is enabled

How It Works

The assistant is designed to operate within constrained napari workflows rather than as a general-purpose chatbot.

The current strategy is:

1. collect structured napari viewer context
2. build deterministic per-layer profile objects from the current viewer state
3. add bounded approved session memory when available
4. send that context and the user request to a local Ollama model
5. the model returns a structured JSON response that specifies either:
   • a normal reply
   • a built-in tool call
   • generated Python code
6. run the selected registry-backed tool or expose the generated code through the UI
7. update session memory from explicit user feedback or successful follow-up behavior

This keeps the assistant more grounded than a plain chat interface and makes common operations more reliable.

Design Direction

The intended architecture is:

1. natural language at the user surface
2. registry-backed tools underneath
3. explicit scope resolution for full-layer and ROI/subregion workflows

This means users should be able to ask for operations in normal language, while the plugin resolves those requests into deterministic tool calls with structured parameters.

Tool model

Registered tools are the common execution model for:

• chat-triggered actions
• reusable UI actions
• future workflow and pipeline steps
• future plugin-contributed extensions

Each tool is moving toward a shared definition with:

• stable name
• parameter schema
• supported layer types
• prepare/execute/apply lifecycle
• UI metadata
• provenance metadata

Scope model

For imaging analysis, operations may target:

• the full layer
• a labels mask
• a specific labels object
• a shapes ROI
• a bounding-box crop

Natural language can express these requests, but the plugin still needs deterministic binding rules underneath.

The preferred resolution order is:

1. explicit user binding such as image_a using roi_shapes
2. current viewer selection when there is only one clear match
3. a short clarification question when multiple bindings are plausible

Session memory should remain secondary context. Current viewer state and explicit user clarification should remain the primary source of truth.

Recommended Models

Good starting choices:

• nemotron-cascade-2:30b
• qwen3-coder-next:latest
• qwen3.5
• qwen2.5:7b

Selection guidance:

• nemotron-cascade-2:30b is the current default and a strong general model for this workflow.
• qwen3-coder-next:latest is a better candidate for Python and napari code generation, but it is significantly heavier.
• qwen3.5 remains a useful alternative general model.
• qwen2.5:7b is lighter and may fit smaller-memory systems more easily.

Memory note:

• Larger tags require more RAM or VRAM.
• On the DGX Spark setup used during development, qwen3-coder-next:latest may need around 100 GB of available memory to run comfortably.

Current Limitations

• the dataset profiler is still Phase 1 and remains strongest on already-loaded napari layers rather than reader- or file-format-specific workflows
• TIFF vs OME-Zarr adapter behavior is not implemented yet
• ND2 and Zeiss reader-aware adapters are not implemented in this plugin
• the tool registry is in progress; some tools are now registry-backed, but the migration is not complete yet
• session memory is selective and bounded; it is not full conversation memory
• model output can still be inconsistent, especially when falling back to generated code
• some requests still miss built-in tools and fall through to code generation when a stronger built-in workflow would be preferable
• generated code can still fail if the model invents incorrect napari APIs or unsupported imports
• multi-step workflow planning and replay are not implemented yet
• no image attachment or multimodal input pipeline yet
• performance optimization for very large 2D/3D datasets is still in progress
• hard native crashes in Qt/C-extension code may not be captured cleanly by the plugin crash log even when normal plugin errors are logged

Most reliable current workflow:

• use built-in tools for common layer inspection and mask/image actions
• trust current viewer context and current layer profiles over any remembered prior turn
• use the Library for repeated prompts, demo packs, and reusable code
• use generated code when you want explicit review and control
• use Run My Code when you already have working Python and want to test it directly inside napari

For demo and education workflows:

• ask for code that uses the current napari viewer
• avoid prompts that create a second napari.Viewer() or call napari.run()
• prefer docked widgets over unmanaged popup windows for histogram or SNR teaching tools

Troubleshooting

Ollama not running

If Test fails after restarting your computer, Ollama is usually not running yet.

Start it in a terminal:

ollama serve

Then return to the plugin and click Test again.

Pulling a model

Model downloads are intentionally handled outside the plugin.

To try a different model:

• browse tags at https://ollama.com/search
• type the tag into the plugin Model field if needed
• pull it in a terminal, for example:

ollama pull nemotron-cascade-2:30b

Then use Test to refresh the plugin state.

Logs and crash logs

The plugin writes two local log files:

• ~/.napari-chat-assistant/assistant.log
• ~/.napari-chat-assistant/crash.log

Use these together with the terminal traceback when diagnosing crashes or unclear UI failures.

Local model telemetry

The plugin also writes lightweight local telemetry to:

• ~/.napari-chat-assistant/model_telemetry.jsonl

This records real usage events such as:

• request start and completion
• selected model and prompt hash
• total latency
• response type (reply, tool, code, or error)
• reject feedback from 👎 Reject
• approved code execution success or failure

Telemetry is now opt-in from the Session -> Telemetry tab through Enable Telemetry.

The goal is passive model tracking during actual work rather than separate benchmarking runs.

For advanced users, the Session -> Telemetry tab includes:

• Summary to generate a quick in-app summary of recent model speed and behavior
• Log to inspect the latest raw JSONL records together with the summary
• Reset to clear the local telemetry file and start fresh from the next request

This keeps the append-only log intact while making it easier to review without leaving napari.

Generated code is also preflight-validated before execution for common dtype mistakes, unsupported napari imports, and unavailable viewer.* APIs. When validation blocks execution, the code remains visible and copyable for review or regeneration.

Development

Editable install:

pip install -e .

Build a release artifact:

python -m build

License

MIT.