napari-zenapi-streaming

napari-zenapi-streaming is a napari plugin and Python app that streams live microscopy image data from ZEISS ZEN Blue into napari via the ZEN API (gRPC).

It supports three workflows selectable from the plugin UI:

• Display: watch ZEN experiment – passively displays an experiment started in ZEN Blue. Channel choices are All or 1-16, independent of the selected setup; its expected frame count is unknown.
• Display: start selected experiment – starts the selected setup from napari and displays its frames. Channel choices match the setup's XML channel count.
• OME-ZARR only – writes frames directly to an OME-ZARR file on-the-fly using the standalone writer.

Display workflows use the same live reader and restructure received frames into per-channel scene mosaics. Channel labels are 1-based; the ZEN API filter uses zero-based indices. The filter defaults to All channels when a previously selected channel is not available in the new mode or experiment.

A standalone CLI (ZEN_stream2omezarr.py) is also available to stream directly to OME-ZARR without napari.

Current status

• Project phase: alpha / MVP
• Primary OS target: Windows for ZEN Blue and the ZEN API Gateway
• Python target: >=3.12, <3.15

Known limitations

• ZEN Blue and the ZEN API Gateway are required for live acquisition. Automated tests use simulated responses; they do not establish end-to-end throughput or frame completeness on real hardware. ZEN Blue and its gateway run on Windows.
• Running napari on a separate Linux host may be possible, but this remote topology is experimental and not officially supported. It requires a fast, reliable connection to the Windows gateway and expertise configuring gateway certificates and trust for the remote client; it has not been validated here.
• The OME-ZARR writer accepts only 2D GRAY8 and GRAY16 frames. Color pixel types are not supported by that output path.
• ZEN-started Display mode monitors the global pixel stream without a known experiment ID; run only one acquisition at a time in this mode.
• Display mode retains received frames in memory for final scene assembly. Very large or long acquisitions may exceed available RAM; the standalone unconfigured --experiment CLI mode also buffers the acquisition.
• Out-of-order OME-ZARR frames use a bounded pending buffer. If the stream stays incomplete until the inactivity timeout, missing positions are skipped rather than recovered from ZEN. The configured writer retains the partial store, writes acquisition_status.json, and reports failure instead of success. Inspect the report before using partial output.
• Display mode currently assumes uint16 input and normalizes every frame for preview. Its restructured layers are not raw quantitative image data. Use configured OME-ZARR output to retain raw GRAY8/GRAY16 intensities.

Features

• Live frame ingestion through ExperimentStreamingService (gRPC).
• Display mode: Buffered async pipeline (reader → queue → processor) for smooth UI updates, sparse frame storage, automatic post-acquisition restructure from a 7D streaming layer to 5D S,T,Z,Y,X scene mosaics per channel (merging the M tiles spatially).
• OME-ZARR mode: Dedicated gRPC channel, tight read loop, on-the-fly writes via ome-writers. Identical code path for both plugin and CLI.
• Acquisition dimensions and Z spacing are derived automatically from the selected experiment XML. Changing the selection refreshes these values.
• OME-ZARR dtype is derived from the runtime frame PixelType; the raw payload size is checked before decoding.
• Incoming frames are assigned from live stream metadata. In particular, FramePosition.s and FramePosition.m select the scene and tile; frame arrival order and XML TileRegion order are not used for assignment.
• Thread-safe napari updates via Qt signals/slots.
• Configurable startup and timeout behavior from .env or CLI.

Repository layout

src/napari_zen_streaming/
  main.py              # App entrypoint and orchestration
  ZEN_config.py        # Runtime config dataclasses + env parsing
  ZEN_init.py          # gRPC connection + experiment lifecycle helpers
  ZEN_pipeline.py      # Async reader/processor pipeline (Display mode)
  ZEN_ui.py            # Viewer UI, layer lifecycle, mode switching
  ZEN_utils.py         # Metadata + image processing functions
  ZEN_omezarr.py       # OME-ZARR helpers, ExperimentConfig, viewer launchers
  ZEN_stream2omezarr.py # Standalone OME-ZARR streaming (CLI + plugin backend)
  _widget.py           # Napari plugin widget entrypoint
  misc.py              # gRPC channel initialization from config.ini
  napari.yaml          # Plugin manifest
  .env                 # Editable-checkout fallback only; not packaged

experiment_streaming_config.ini  # Experiment config for OME-ZARR streaming

docs/
  *.mmd               # Mermaid flowchart and sequence sources
  *.png               # Rendered diagram preview

Requirements

• ZEISS ZEN Blue with ZEN API Gateway installed
• A valid ZEN control token
• Access to the ZEN API certificate (ZenApiPersonalSigningRootCA.pem)
• Python 3.12-3.14 with Qt >=6.7; PyQt6 is used in CI. PyQt5 is not compatible with napari 0.10 or newer.
• Authorized, compatible zen_api and zen_api_utils wheels from ZEISS OAD (neither package is distributed on PyPI)

Installation

Required before a normal PyPI installation: zen_api and zen_api_utils are required dependencies, but pip cannot discover their GitHub wheels on PyPI. On a fresh environment, pip install napari-zenapi-streaming will fail with a resolver error such as No matching distribution found for zen_api. Install both compatible wheels as shown below, then retry the plugin install. This happens before plugin code runs, so the plugin cannot display a custom installation error or GUI message.

Install the required ZEN wheels

Use the same Python environment for the wheels, the plugin, and napari. The current dependency requirements are:

• zen_api>=2026.5.1,<=2026.11.1: choose a version compatible with your ZEN API Gateway.
• zen_api_utils>=0.1.0,<0.2: utilities have their own version number, separate from the API release.

For example, install API 2026.7.1 and utilities 0.1.0 from OAD:

$base = "https://github.com/zeiss-microscopy/OAD/releases/download"
python -m pip install `
  "$base/v2026.07.1/zen_api-2026.7.1-py3-none-any.whl" `
  "$base/zen-api-utils-v0.1.0/zen_api_utils-0.1.0-py3-none-any.whl"

The backtick is PowerShell's line-continuation character. Alternatively, pass the paths to your downloaded authorized wheels:

python -m pip install `
  "C:\path\to\authorized\zen_api-<version>-py3-none-any.whl" `
  "C:\path\to\authorized\zen_api_utils-0.1.0-py3-none-any.whl"

Replace the example paths and API version with the wheels supplied for your gateway. Do not commit wheels, tokens, or certificates to this repository without confirming redistribution rights.

Install the plugin

For a published PyPI release:

python -m pip install napari-zenapi-streaming "napari[pyqt6]" "PyQt6>=6.7,<6.10"

From this checkout, install the plugin:

python -m pip install .

For a downloaded release wheel, pass that wheel path instead of .. For development use python -m pip install -e ".[dev,testing]".

If multiple Qt bindings are installed, select PyQt6 before launching napari:

$env:QT_API = "pyqt6"

The two ZEN wheel URLs or paths can also be supplied alongside the plugin in one python -m pip install command; installing them beforehand is not required when they are provided explicitly. Keep the required version constraints in the plugin metadata rather than using --no-deps to bypass them.

If pip still reports a missing ZEN dependency, confirm that both wheels satisfy the version ranges above and were installed with the same python executable used for the plugin. Run python -m pip check after installation to check for missing or incompatible dependencies.

Conda environment

Alternatively, create the supplied Conda environment first, then install the two authorized wheels and this plugin as above:

conda env create -f python_env/env_zenapi_napari.yml
conda activate zenapi-napari

The Conda manifest intentionally does not download the ZEN wheels; supplying both zen_api and zen_api_utils is required for both installation routes.

Configuration

1) config.ini (required)

Create a user-owned gateway configuration:

python -m napari_zen_streaming.ZEN_config --init

The command prints the new path and never overwrites an existing file. Edit the host, certificate path, and control token before opening the plugin. The default directory is %APPDATA%\napari-zenapi-streaming on Windows and $XDG_CONFIG_HOME/napari-zenapi-streaming (or ~/.config/napari-zenapi-streaming) elsewhere. Configuration does not belong inside site-packages.

Set ZEN_CONFIG_FILE to use an existing gateway INI instead. The standalone app also accepts --config-file. Explicit relative paths resolve from the working directory. For source-checkout compatibility, an existing config.ini in the working directory or checkout root is used when no user configuration exists. User configuration takes priority over these fallbacks.

[api]
host = 127.0.0.1
port = 5000
cert_file = C:\ProgramData\Carl Zeiss\ZenApiGateway\Certificates\ZenApiPersonalSigningRootCA.pem
control-token = your-control-token

[image_streaming]
host = 127.0.0.1
port = 5280

The loopback addresses above assume napari runs on the same Windows host as ZEN Blue. A remote client needs reachable control and image-streaming hosts and a certificate configuration appropriate to the gateway hostname. The certificate path shown above is a Windows-local example, not a Linux path.

2) Environment variables (optional)

Define settings in the process environment or a .env beside the user-owned config.ini. Set ZEN_ENV_FILE to load a different dotenv file; an explicit missing file is an error. Editable checkouts also retain the package-local .env as a fallback when no user settings file exists.

• ZEN_EXP_NAME – initially selected experiment setup. Default: ZEN_API_overview.
• ZEN_UI_START – true initially selects napari-started Display; false selects passive ZEN watch. Default: true.
• ZEN_CONFIG_FILE – path to the gateway configuration. Default: user directory.
• ZEN_ENV_FILE – explicit optional settings file. Default: user .env.
• ZEN_CHANNEL_INDEX – gRPC channel index filter (integer). Leave empty or unset to receive all channels.
• ZEN_RESTRUCTURE_TIMEOUT – idle timeout in seconds before final restructure in ZEN-started mode. Default: 10.0.
• ZEN_SHOW_ONLY_LAST_FRAME_DURING_LIVE – use the lightweight 2D latest-frame preview in Display mode. Default: true.
• ZEN_OUTPUT_DIR – default output directory for OME-ZARR files. Default: unset.
• ZEN_LOG_DIR – directory containing the rotating zen_streaming.log file. Default: platform-specific, see Logging.
• ZEN_LOG_LEVEL – DEBUG, INFO, WARNING, ERROR, or CRITICAL. Default: INFO.

The optional .env supplies runtime settings such as the experiment name, startup mode, channel filter, display behavior, and the path to config.ini. It does not contain the ZEN API control token or certificate; those remain in the required config.ini.

Both plugin and CLI preserve existing process environment values. Precedence is explicit CLI flags (where supported), process environment, .env, then code defaults. Only the selected dotenv file is loaded; files are not merged.

To place the logfile in a custom directory, set an absolute path in the user .env, for example:

ZEN_LOG_DIR=C:\Microscopy\logs
ZEN_LOG_LEVEL=DEBUG

Leave ZEN_LOG_DIR= empty or remove the line to use the platform-specific default documented in Logging. Restart napari after changing the file because logging is configured when the plugin widget opens.

Default OME-ZARR output directory

Set the plugin's initial output folder with ZEN_OUTPUT_DIR in the user settings file. On Windows this is %APPDATA%\napari-zenapi-streaming\.env. The settings directory may not exist yet; create it and open the file from PowerShell:

$settingsDir = Join-Path $env:APPDATA "napari-zenapi-streaming"
New-Item -ItemType Directory -Force $settingsDir | Out-Null
notepad (Join-Path $settingsDir ".env")

If Notepad asks to create the file, accept. Add or update the following entry without removing any existing settings, then save it with the exact filename .env, not .env.txt:

ZEN_OUTPUT_DIR=F:\Zen_Output

Creating this .env does not require creating a user config.ini. If your gateway configuration already works, leave it in place. Do not create an empty config.ini in the user settings directory: it takes priority over working directory and checkout fallbacks. For a new gateway configuration, use the configuration initializer and populate its template before connecting.

The user .env replaces the editable checkout's fallback .env; the files are not merged. Include any checkout settings you still need, such as ZEN_CONFIG_FILE. Existing process environment variables override .env values. Restart napari after changing the file.

Alternatively, set the output default for one PowerShell session before launching napari:

$env:ZEN_OUTPUT_DIR = "F:\Zen_Output"
napari

The folder field in the plugin overrides this initial value. Without ZEN_OUTPUT_DIR, it defaults to your Documents directory. The standalone experiment INI's [output] directory is separate and does not configure the plugin.

Before configured streaming starts an experiment or opens pixel streams, the writer creates the output folder if needed and probes temporary-directory and file creation, then removes the probe. It also rejects the Windows read-only attribute because the native writer rejects such parent folders even when Python can write to them. Windows Documents may have that attribute; choose a writable subfolder or another output directory instead. This preflight does not guarantee sufficient disk space or that access will remain available.

Running

Option A: Run as a napari plugin

1. Start napari:

   napari
   

2. Open the plugin widget: Plugins > ZEN API Streaming > ZEN API Streaming.

Option B: Run as a module

python -m napari_zen_streaming.main

With CLI overrides:

python -m napari_zen_streaming.main --ui-start true \
    --exp-name ZEN_API_overview --config-file config.ini \
    --channel-index 0 --restructure-timeout 10

Runtime behavior

Operating modes

The plugin UI exposes three stream modes. Connecting the plugin never starts an acquisition; only clicking Start in a start-capable mode does so.

Display: watch ZEN experiment (index 0)

The plugin waits for an experiment started independently in ZEN Blue. No selected setup or known frame count is assumed, and there is no Start button. The channel filter offers All channels or labels 1-16 (ZEN indices 0-15).

Display: start selected experiment (index 1)

The Start button loads and triggers the selected experiment via the ZEN API. Its XML dimensions are shown in the widget, and the channel filter offers All channels or exactly the channels available in that setup. Until metadata has loaded, only All channels is available. Changing the selected setup reloads both its dimensions and channel choices. Click Update ZEN Experiment Data below the selector to reload the current setup without changing the selection.

Both Display modes use the full async streaming pipeline (ZEN_pipeline.py):

1. Streaming phase – Frames are received via monitor_all_experiments(), processed in ZEN_utils.process_frame(), and stored in a sparse dict. The sparse key (S, T, M, Z, C) comes directly from each frame's FramePosition metadata, including its S and M indices. By default, Show only last frame during live displays a fixed 2D YX layer and replaces it with the newest frame without rebuilding acquisition history. Every frame remains in sparse storage for final assembly. Uncheck the option to restore the more expensive 7D frame-history preview. The option is frozen during acquisition and re-enabled after scene assembly.
2. Channel filter – Changing the filter immediately restarts the pipeline reader. Labels are 1-based, while the gRPC request uses zero-based indices. The initial value comes from ZEN_CHANNEL_INDEX in .env; leave it empty to receive all channels. An out-of-range choice resets to All channels.
3. Completion detection –
   • Napari-started experiments: the status monitor watches is_experiment_running. Because ZEN can report completion before all gRPC pixel data arrives, the plugin then waits for the XML-derived unique frame count and uses pixel-stream inactivity as a safety fallback.
   • ZEN-started experiments: inactivity timeout (ZEN_RESTRUCTURE_TIMEOUT).
4. Post-processing – The streaming layer is removed and each scene's M tiles are placed from their streamed stage coordinates into per-channel 5D layers (S, T, Z, Y, X). In overlap regions, pixels from the tile with the higher M index replace pixels from lower-M tiles.

Experiment identity: Display mode keeps one monitor_all_experiments() subscription open because the targeted ZEN stream can close when experiment status changes to finished, before trailing pixel payloads have arrived. For a Napari-started acquisition, the plugin loads the experiment first, arms a client-side filter for its FrameData.experiment_id, and only then starts acquisition. Frames from other experiments are ignored while that filter is active, including during trailing-frame drain. The filter is cleared after final restructure so later ZEN-started acquisitions remain discoverable. A ZEN-started run has no experiment ID known in advance and therefore continues to accept the global stream; keep only one such acquisition active at a time.

OME-ZARR only (index 2)

Bypasses the pipeline entirely. When Start is clicked:

1. A fresh ExperimentConfig is built from the selected experiment, the OME-ZARR controls, the main ZEN API gateway config, and fixed plugin defaults. XML-derived T/C/Z/M/S and Z spacing replace its planning placeholders before writing. No experiment INI is selected by the plugin.
2. Auto-trigger checkbox — "Start experiment via ZEN API":
   • Checked (default): clicking Start immediately triggers the experiment via the ZEN API and calls stream_to_omezarr_with_config(ecfg).
   • Unchecked: clicking Start arms the writer (button shows "Writer Armed — start ZEN Experiment") and waits for the user to start the experiment manually in ZEN Blue. The writer activates as soon as the first frame arrives.
3. stream_to_omezarr_with_config(ecfg) — the same function used by the standalone CLI — creates its own dedicated gRPC channel, opens the pixel stream, optionally starts the experiment, and queries live status when the first frame arrives. Positive live T/C/Z/S/M counts define the valid coordinate ranges; XML-derived values remain the fallback if status is unavailable.
4. Each frame is routed by (T, S, M, C, Z). T/S/M/Z come from that frame's FramePosition; C comes from the explicit channel subscription because all-channel mode merges one gRPC stream per source channel. The tuple is mapped to the linear append order required by ome-writers. Frames may arrive in any order and wait in a pending buffer. Duplicate tuples and tuples outside the expected live/configured ranges are skipped; only consecutive linear slots are appended to the Zarr stream. The first frame's runtime PixelType selects uint8 for GRAY8 or uint16 for GRAY16; payload length is validated before decoding. ZEN width is stored on X and height on Y without a rotation or transpose. Streamed X/Y stage positions and pixel scales retain the same axis names. Auto-triggered runs accept only their returned experiment ID. Manually armed runs pin to the first received experiment ID; ensure that first frame belongs to the intended acquisition. Later unrelated frames are ignored before decoding and frames without an experiment ID are rejected.
5. The loop ends after every unique expected coordinate has arrived, or on inactivity timeout. On timeout, buffered frames are drained and truly missing slots are emitted with stream.skip() so the array stays rectangular. The configured writer records expected/received counts and missing (t, s, m, c, z) coordinates in acquisition_status.json inside the output store. Missing frames raise IncompleteAcquisitionError; the CLI exits nonzero and the plugin shows a warning containing the partial output path. Partial stores are retained, but are not auto-opened and do not receive post-acquisition pyramids. Other writer errors also fail the run; a status report is guaranteed only after successful finalization.
6. For generic acquisitions with multiple scenes or tiles, the raw tile streams are finalized into one TCZYX mosaic per scene. Streamed XYZ coordinates and physical pixel scaling determine placement; M identifies the raw tile within S but does not imply a row or column. In overlaps, the highest M index wins, and gaps in irregular scene outlines remain at the array fill value. The OME/series list advertises only the scene mosaics. Keep individual source tiles controls whether the raw tile groups remain after successful mosaic assembly. It is unchecked by default to avoid duplicated pixel storage. For HCS acquisitions, one frame from every M tile first establishes each scene canvas. The writer then buffers only the tiles for each (T, S, C, Z) plane, merges them in memory, and writes the final well image directly. Raw HCS tile arrays are never created, so the store is not reread or rewritten after acquisition. The source-tile control is hidden for HCS output because it does not apply.
7. Button state sequence: Start Selected Experiment → (click) → Writer Armed — start ZEN Experiment (auto-trigger off) or Experiment Running... (auto-trigger on) → Experiment & Stream Running (first frame arrived) → back to Start Selected Experiment (done).
8. Worker-to-Qt progress notifications are limited to approximately 10 Hz. This keeps napari responsive without throttling frame reception or OME-ZARR writes; first-frame, final-frame, and completion updates are always delivered.

The plugin and the configured CLI share the same writer. Both use a dedicated gRPC channel while the display pipeline reader is suspended, avoiding frame loss from competing consumers. The unconfigured --experiment CLI mode instead buffers the full acquisition. That legacy buffered mode cannot certify a complete expected coordinate set; use configured mode for acquisition completeness reporting.

Starting a new acquisition removes only layers created or opened by this plugin. Unrelated user layers and other plugins' dock widgets are retained.

Plugin OME-ZARR defaults

The plugin exposes output directory, compression, auto-trigger, HCS layout, and Create pyramid after acquisition in its UI. When enabled, choose how many 2x Y/X downsampled levels to append after acquisition completes. Napari opens the OME-Zarr only after pyramid generation finishes. Existing level 0 is reused and is never rewritten. Generic scene mosaics also expose Keep individual source tiles; unchecked removes source groups only after mosaic creation succeeds. Advanced values intentionally use fixed defaults declared at the top of ZEN_omezarr.py:

• PLUGIN_DEFAULT_CZI_NAME = "zenapi_stream"
• PLUGIN_DEFAULT_OVERWRITE_CZI = True
• PLUGIN_DEFAULT_OVERWRITE_ZARR = True
• PLUGIN_DEFAULT_SPATIAL_SHARD_SIZE_CHUNKS = 2
• PLUGIN_DEFAULT_PYRAMID_LEVELS = 0

Change those constants to alter plugin behavior globally. Plugin OME-ZARR dtype is not a fixed default: it is derived from the first runtime frame's ZEN PixelType. Color pixel types are rejected because the current writer models each streamed frame as a two-dimensional grayscale plane.

Standalone experiment config INI

The standalone CLI config mode accepts experiment_streaming_config.ini. The plugin does not load this file. Before opening pixel subscriptions, the CLI loads and exports the named experiment through ZEN API. Exported T/C/Z/M/S, Z spacing, and scene-to-well positions replace the INI's dimension defaults; this ensures all active channels are subscribed and explicit wells can use the HCS plate layout. The INI supplies the other run settings:

• [experiment] – name, CZI output name, start mode, overwrite settings.
• [dimensions] – legacy dimension fallbacks for callers of load_experiment_config(). The configured CLI replaces them with exported experiment metadata before opening the writer.
• [output] – directory, dtype fallback, compression, overwrite, keep_source_tiles, and optional spatial_shard_size_chunks for Zarr v3 Y/X sharding. Set the shard size to 0 or leave it empty to disable sharding. pyramid_levels controls post-acquisition Y/X pyramid creation.
• [stream] – optional source channel filter and max_pending_frames safety bound for out-of-order frames.

Before OME-ZARR streaming, both the plugin and the configured CLI load and export the selected ZEN experiment to derive T, C, Z, M, and S from its XML. Active rectangular TileRegion grids describe scene-specific tile layouts; for example, two active 3-by-2 grids produce S=2 and M=6. The total frame count is recalculated from the exported metadata. These values plan the acquisition and provide a fallback, but they do not assign incoming pixels. The first live status snapshot replaces positive T/C/Z/S/M counts before the writer layout is opened. Every frame is then assigned by its live FramePosition, including FramePosition.s and FramePosition.m; per-frame XYZ and scaling remain authoritative for physical tile placement.

Stream metadata authority

The writer deliberately separates acquisition planning from frame routing:

• Selected experiment XML – planned T/C/Z/S/M and Z spacing; fallback when live status is unavailable.
• First live get_status() response – positive T/C/Z/S/M counts used to build the valid coordinate space and expected frame set.
• Per-frame FramePosition – T/S/M/Z assignment for that incoming image. C is the merged subscription index.
• First frame pixel_data.pixel_type – OME-ZARR NumPy dtype, validated against the raw payload byte count.
• Per-frame stage XYZ and pixel scaling – native X/Y physical placement and scale of each M tile inside its S scene.

Consequently, ZEN may deliver channel-major or otherwise out-of-order frames without changing their destination. Scene membership is always the streamed S index, tile identity is always the streamed M index, and spatial placement is always based on streamed physical coordinates rather than an assumed grid.

Standalone CLI

For a Python API example, see examples/stream_to_omezarr.py. It loads the experiment INI, exports acquisition metadata, and awaits stream_to_omezarr_with_config(). Use the same environment and gateway configuration described above. From the repository root:

python examples/stream_to_omezarr.py

With no argument, it uses experiment_streaming_config.ini from the repository root, located relative to the script rather than the working directory. Pass another INI path as the positional argument to override this default.

Use --experiment NAME to override the INI's experiment name for both metadata export and streaming. It does not start acquisition unless you also pass --start-experiment:

python examples/stream_to_omezarr.py --experiment "MyExperiment" --start-experiment

Edit the INI's experiment name and output directory before running. By default, the example waits for you to start the acquisition in ZEN Blue. Add --start-experiment to trigger it via the API. The example disables CZI and OME-ZARR overwriting, honors the INI's pyramid settings, and lets errors propagate so incomplete acquisitions exit nonzero and retain partial output.

ZEN_stream2omezarr.py can be run directly without napari:

# Config mode (export metadata first, then write on-the-fly):
python -m napari_zen_streaming.ZEN_stream2omezarr \
    --experiment-config experiment_streaming_config.ini

# CLI mode (unknown dimensions, buffered write):
python -m napari_zen_streaming.ZEN_stream2omezarr \
    --experiment MyExp --output-dir ./output --start-experiment

# Wait for user to start from ZEN UI:
python -m napari_zen_streaming.ZEN_stream2omezarr \
    --experiment-config experiment_streaming_config.ini --no-start-experiment

# Open viewer after acquisition:
python -m napari_zen_streaming.ZEN_stream2omezarr \
    --experiment-config experiment_streaming_config.ini --viewer ndv

# Override the INI and append three pyramid levels after acquisition:
python -m napari_zen_streaming.ZEN_stream2omezarr \
  --experiment-config experiment_streaming_config.ini --pyramid-levels 3

Logging

• Log file on Windows: C:\Users\<username>\AppData\Local\napari-zenapi-streaming\logs\zen_streaming.log (%LOCALAPPDATA%\napari-zenapi-streaming\logs\zen_streaming.log). AppData is hidden by default in File Explorer.
• Log file on Linux/macOS: $XDG_STATE_HOME/napari-zenapi-streaming/logs/zen_streaming.log (or ~/.local/state/napari-zenapi-streaming/logs/zen_streaming.log)
• Set ZEN_LOG_DIR to override the log directory.
• Logs rotate at 5 MB and retain three backups.
• Messages are also written to the terminal that launched napari after the plugin widget is opened.
• Set ZEN_LOG_LEVEL=DEBUG in .env to include connection, pipeline, frame processing, layer-update, and restructure details. The default INFO level records only milestones, warnings, and errors.
• Logging format: %(asctime)s - %(name)s - %(levelname)s - %(message)s

To see live terminal records, launch napari in the configured environment and keep that terminal open:

napari

Logging starts when the ZEN API Streaming plugin widget is opened. A napari process launched from the Start menu or another detached GUI launcher cannot write records into an existing PowerShell terminal.

On Windows, print the exact logfile location for the current user:

$logFile = "$env:LOCALAPPDATA\napari-zenapi-streaming\logs\zen_streaming.log"
$logFile

Open its directory in File Explorer:

explorer.exe "$env:LOCALAPPDATA\napari-zenapi-streaming\logs"

If ZEN_LOG_DIR is not set, follow the default Windows logfile in a separate PowerShell terminal:

Get-Content "$env:LOCALAPPDATA\napari-zenapi-streaming\logs\zen_streaming.log" -Wait

When ZEN_LOG_DIR=C:\Microscopy\logs, use the configured location instead:

Get-Content "C:\Microscopy\logs\zen_streaming.log" -Wait

Get-Content prints the existing records and -Wait keeps the command running so new records appear as they are written. Press Ctrl+C to stop. If the 5 MB logfile rotates while this command is running, restart the command to follow the new active zen_streaming.log file.

Architecture

The latest-frame preview avoids rebuilding a dense history array during acquisition. Sparse frames remain available for final scene assembly; disabling the preview option restores full-history live display.

Display mode data flow

The global pixel stream feeds the pipeline reader, queue and processor. Qt signals update sparse frame storage and the latest 2D preview, or the optional 7D history layer. For napari-started runs, the experiment ID filter stays armed through the trailing-frame drain after status completion. Final restructure creates one STZYX scene-mosaic layer per channel, then clears the ID filter. Only plugin-owned layers are removed.

OME-ZARR mode data flow

The plugin suspends the Display reader before the shared configured writer opens its dedicated channel and per-source-channel pixel streams. Experiment IDs are checked before decoding. Runtime GRAY8/GRAY16 metadata selects the dtype; positive live dimensions override XML fallback counts. Unique (T,S,M,C,Z) keys drive bounded buffering and ordered writes, either as raw generic tiles or directly merged HCS scene planes.

After completion or inactivity, the writer drains pending slots, finalizes the layout and writes acquisition_status.json. Complete stores may receive pyramids and open in a viewer. Missing coordinates instead raise IncompleteAcquisitionError, preserve partial output and suppress pyramids and auto-opening. The plugin warns on failure and resumes the Display reader in finally; the configured CLI exits nonzero for incomplete acquisitions.

Both flows use the same monitor_all_experiments() gRPC call. The OME-ZARR path creates its own channel so there is no contention.

Diagrams and docs

• Mermaid flowchart
• Mermaid sequence diagram

The diagrams focus on the three plugin workflows and their shared configured writer. The standalone buffered CLI mode is described separately above.

Flowchart preview:

Render the Mermaid sources with Mermaid CLI, or preview .mmd files in the Mermaid VS Code extension. From the repository root:

npx --yes --package @mermaid-js/mermaid-cli -- mmdc \
    -i docs/napari_zenapi_streaming_flowchart.mmd \
    -o docs/napari_zenapi_streaming_flowchart.png

Troubleshooting

• Missing config.ini: run python -m napari_zen_streaming.ZEN_config --init, populate the template, or set ZEN_CONFIG_FILE to an existing file.
• TLS or certificate issues: verify the cert_file path and that the file is readable.
• Qt5 incompatibility: install "PyQt6>=6.7,<6.10" and select QT_API=pyqt6. For pytest, also set PYTEST_QT_API=pyqt6.
• No experiments in dropdown: confirm ZEN API Gateway is running and token is valid.
• Viewer not updating: check zen_streaming.log in the user log directory described above for pipeline errors.
• Restructure too early/late for ZEN-started runs: adjust ZEN_RESTRUCTURE_TIMEOUT.
• Progress bar stuck on "Streaming frames": if ZEN stops an experiment without the plugin detecting it (e.g. network glitch), click Stop to dismiss the bar. Clicking Start on the next experiment also clears any stale bar automatically.
• Incomplete acquisition: inspect acquisition_status.json and the warning's partial-store path. Check frame counts, gateway logs, and inactivity timeout before retrying; a rectangular output array does not imply complete data.
• Channel filter not taking effect: ensure no experiment is currently running; the reader restarts on combo box change, but an in-flight gRPC stream may buffer a few straggler frames.

Development

• Install development extras:

python -m pip install -e ".[dev,testing]" build
python -m pytest -q src/napari_zen_streaming/_tests
python -m ruff check src/napari_zen_streaming
python -m ruff format --check src/napari_zen_streaming
python -m build --sdist --wheel

• Ruff lints and formats the code using the settings in pyproject.toml (line length 79). In VS Code, set "ruff.configurationPreference": "filesystemFirst" so user-level Ruff settings do not override the repository configuration.
• Tests mock the gateway and cover routing, partial acquisitions, mosaics, metadata refresh, dtype handling, progress, configuration, and layer ownership.
• A clean Windows environment passed the full test suite with Python 3.12.15, napari 0.10.0, PyQt6 6.9.1, ome-writers 0.3.2, ngff-zarr 0.51.0, zen-api 2026.7.1, and zen-api-utils 0.1.0. This records one tested environment, not a claim that every gateway version is compatible.
• CI checks Python 3.12-3.14 on Windows, Linux, and macOS. Pushes and pull requests target main; version tags trigger publishing only after tests pass.
• CI supplies the public OAD release wheel URLs for zen_api==2026.7.1 and zen_api_utils==0.1.0 in the same pip command as the plugin installation. No ZEN wheel URL secret is required, including for forked pull requests.
• The testing extra installs PyQt6>=6.7,<6.10. Both CI workflows set QT_API=pyqt6 and PYTEST_QT_API=pyqt6 to avoid selecting another binding.

Before tagging a release

• Confirm the pinned OAD release wheels remain reachable and the full CI matrix passes. Local tests do not replace this check.
• Choose the release version, update pyproject.toml and CHANGELOG.md, and verify the tag matches. No release version is selected by the cleanup.
• Install the built wheel in a clean environment, initialize user configuration, and verify plugin discovery without relying on this checkout.
• On a real ZEN gateway, test watch/start Display modes, manual/auto-triggered OME-ZARR, multiple channels and tiles, HCS wells, and pyramid generation.
• Interrupt a real acquisition: confirm a visible warning, non-success status, preserved partial output, and correct missing-coordinate reporting.
• Verify Stop, disconnect, viewer close, repeated runs, and preservation of unrelated layers and docks. Measure throughput and memory at production size.

Authors

• Krijn H. van der Steen (UMC Utrecht)
• Sebastian Rhode (ZEISS)

License

BSD 3-Clause (see LICENSE).