Driftless Bundle Photometry

Version 0.2.1 · What's new

A Python-based, trace-first multichannel fiber-photometry acquisition system.

The application supports up to nine circular fiber ROIs, interleaved 405/470/565 nm excitation, four edge-recording TTL inputs, optional lossless raw frames, and one self-contained NWB-HDF5 output per ROI and animal. Windows is the physical-hardware target; the simulator and GUI demo are cross-platform.

Current implementation

• Immutable validation for sessions, channels, circular ROIs, camera settings, and TTL inputs.
• Deterministic simulator and a shared headless acquisition engine with explicit lifecycle states.
• Bounded, checksummed, chunked session spools that survive interruption and disk or finalizer errors.
• Pre-arm storage-capacity checks and structured live diagnostics for queue pressure, write latency, dropped frames, clock residuals, and NWB finalization stages.
• Typed acquisition faults are committed as system events and invalid-time markers before an incomplete spool closes whenever committed samples can be recovered.
• Canonical NWB finalization using PyNWB, ndx-fiber-photometry, and ndx-ophys-devices, including a separate subject, brain region, sensor type, and validated file for every ROI.
• Exact per-exposure commanded voltage, controller sequence/tick, camera frame ID, host receipt time, per-ROI saturation QC, explicit excitation events, and marked invalid continuous spans in every independently interpretable animal file.
• Trace-first PySide6/pyqtgraph desktop demo with a separate calibration image and draggable circular fiber ROIs, plus separate live camera panels for 405, 470, and 565 nm exposures.
• Selectable 15 s, 1 min, 10 min, 1 h, and Full trace horizons with display-only extrema-preserving decimation, keeping long recordings responsive without changing acquired or stored samples.
• One live plot row per ROI with overlaid 405, 470, and 565 nm traces, independent wavelength visibility controls, and Absolute or display-only dF/F views with a configurable per-ROI/per-wavelength median baseline window.
• Optional display-only trailing-mean smoothing uses elapsed seconds, works with asynchronously sampled channels, and never changes acquired or stored values.
• Independent enable/disable controls for each ROI; disabled definitions remain in saved settings but are excluded from extraction, live traces, and NWB output.
• A required non-writing acquisition preview that validates timing, explicit wavelength coverage, commanded voltages, camera frames, ROI extraction, and saturation before enabling each recording.
• Synchronized numeric time axes across all ROI rows, horizontal/vertical drag-box zoom, double-click zoom reset, and page scrolling—not graph zoom—under the mouse wheel.
• Clean numeric entry fields without embedded increment/decrement stepper buttons.
• Versioned, human-readable JSON settings with GUI actions to save, load, set startup defaults, and restore the complete setup from a previous DBF NWB recording.
• Settings v2 discriminate the native camera/controller source from the planned RWD read-only bridge and migrate v1 files explicitly to native with a visible warning.
• A raw uint16 reference frame and fixed-scale ROI-annotated diagnostic view for every wavelength observed in a recording are saved inside each ROI NWB file.
• Versioned/checksummed controller protocol, fail-safe lifecycle guard, and single-owner serial transport that sends a best-effort STOP on close.
• Lazy Thorlabs SDK detection that does not break simulator mode on macOS.

The CS505MU camera transport, serial controller transport, controller firmware, and LED intensity electronics are not yet physically validated. Those require the real Windows rig, vendor SDK, trigger cabling, DAC/driver hardware, and bench timing and safety measurements. Simulator success is not evidence of physical validation.

The RWD bridge currently has a documented wire contract, validated source settings, typed raw domain records, a fixture-tested incremental fluorescence/event parser, and a synthetic-data trace-only recovery/NWB path. The RWD spool retains checksummed mixed records, exact raw ticks/values and scales, named ON/OFF events, connection identity, faults, and invalid spans. Recovery creates one independently validated NWB per mapped fiber without inventing camera frames, LED commands, native exposures, or numbered TTL inputs. A headless single-owner TCP client now provides bounded recording, timeout/disconnect/malformed-stream faults, prompt stop, storage preflight, and machine-readable CLI output. The GUI workflow is still being implemented, and the desktop setup now begins with a native/simulator or RWD read-only system choice, hides camera/LED controls in RWD mode, retains per-fiber animal metadata, and routes both sources through the same bounded live-trace presentation contract. Every RWD fiber has an explicit device-channel number and an independent operator label. Live status reports connection state, decoder counts, spool pressure, committed records, and named event activity; raw, elapsed-time-smoothed, and dF/F views remain presentation-only. The TCP path has only synthetic local-server validation; current builds must not be described as a physically validated RWD connection. See the RWD streaming contract and the RWD operator and bench-validation guide.

The historical MATLAB, Arduino, and Bonsai sources remain under old mescoscope/ for reference and are not runtime dependencies.

Saving and restoring settings

The Configuration panel provides Save JSON, Load JSON, Load NWB, and Set as default. A settings file includes all session metadata, recording duration, output location, native/RWD source selection, source-specific connection settings, camera and retention settings, excitation states and voltages, TTL configuration, live-display preferences, and every fiber's animal metadata. Settings v1 files are accepted only through an explicit migration to the native source because they predate system selection.

On Windows, DBF resolves the same Documents folder used by Explorer, including OneDrive or other redirected locations, and uses its Driftless Bundle Photometry folder for settings. Set as default writes default_settings.json there; DBF loads that file automatically at startup. Save and Load dialogs open in the same folder by default.

Every newly recorded per-ROI NWB file embeds the complete settings snapshot, so any one of those files can restore the whole multi-ROI setup. DBF can also import older per-ROI DBF NWBs: it gathers matching sibling files to recover all available ROIs and warns when an old file never recorded a setting that must use a safe default. Each file also embeds a separate runtime provenance snapshot with the application, Python, operating-system, dependency, adapter, and protocol versions that governed acquisition or recovery.

Installation choices

End users on 64-bit Windows should use the standalone installer. Developers should use the repository's Conda environment. PyPI is also available for users who already manage Python environments. The standalone installer includes Python and all normal application dependencies; it does not require Conda or a separate Python install.

Windows installer for end users

1. Open the latest GitHub release.
2. Download Driftless-Bundle-Photometry-<version>-Windows-x64-Setup.exe.
3. Optionally compare its SHA-256 digest with SHA256SUMS.txt from the same release.
4. Run the installer, then launch Driftless Bundle Photometry from the Start menu.

The installer is per-user, supports clean uninstall through Windows Settings, and does not require administrator access. Current installers are not code-signed, so Windows SmartScreen may show an unrecognized-publisher warning. The application can run the simulator and RWD read-only path, but the physical Thorlabs/controller path must not be treated as validated until the documented Windows bench gates pass.

Conda environment for developers

Clone the repository and let environment.yml create the dbf environment plus an editable install with GUI, test, lint, build, and release tools:

git clone https://github.com/mohebi-n-associates/DriftlessBundlePhotometry.git
cd DriftlessBundlePhotometry
conda env create --file environment.yml
conda activate dbf
python -m pytest

After pulling dependency changes, synchronize the existing environment with:

conda env update --file environment.yml --prune

PyPI installation

Inside an existing Python 3.11–3.13 environment, install the desktop application from PyPI and launch it with:

python -m pip install "driftless-bundle-photometry[gui]"
dbf --version
dbf --demo

The distribution name is driftless-bundle-photometry, the short command and Conda environment are dbf, and Python imports use driftless_photometry. The long driftless-photometry command remains available. On macOS, use Conda or PyPI for the simulator, replay, NWB, analysis, and GUI-demo paths.

Running developer workflows

Run a headless simulated acquisition:

dbf --headless --duration 5 --raw

Run the headless read-only RWD bridge from a validated settings-v2 file:

dbf --rwd-settings path/to/rwd.settings.json --duration 60

RWD must already be recording and listening on the host/port stored in that file. DBF does not send hardware commands. It makes one connection for one session and never reconnects automatically after a fault, because joining records across TCP connections could hide a gap or changed device state.

Capture a checksummed exact-wire artifact during a physical bench run, inspect it, and replay it unpaced through the production parser/recovery/NWB path:

dbf --rwd-settings path/to/rwd.settings.json --duration 300 \
    --rwd-wire-capture path/to/bench-001.rwd-wire
dbf --inspect-rwd-capture path/to/bench-001.rwd-wire
dbf --rwd-settings path/to/replay.settings.json \
    --rwd-replay path/to/bench-001.rwd-wire

Replay settings must preserve the capture's parsing scales, wavelengths, expected machine, and channel/fiber mappings, but should use a new session ID and empty output directory. Exact-wire captures can contain experimental data and must not be committed to the repository.

Run the GUI demo:

dbf --demo

Recover all committed data from a complete or interrupted spool:

dbf --recover-spool path/to/session.photometry-spool
dbf --recover-spool path/to/session.rwd-spool

Inspect direct child spools without creating or removing files:

dbf --inspect-spools path/to/output-directory

Inspection recognizes native *.photometry-spool and RWD *.rwd-spool children, verifies checksums, and reports source-specific counts plus completeness, invalid times, system events, schema, and output-file count. It deliberately does not recurse into unrelated directories. Successful recovery validates the complete NWB set before removing the spool; add --keep-spool for forensic retention. Recovery is idempotent and can resume if a process stopped after validating partials or while promoting the multi-ROI set. Existing canonical files must match the spool exactly and are never overwritten silently. Incomplete acquisitions are explicitly marked in the NWB system event table. Legacy schema-v1 spools remain recoverable, but the resulting invalid-time record identifies the per-frame host receipt and commanded intensity provenance that those older spools never persisted.

The storage estimate uses configured duration, total frame rate, image dimensions, raw-retention policy, enabled ROI count, and a 64 MiB reserve. CI correctness tests exercise non-realtime simulator sessions at 300 frames/s trace-only and 60 frames/s with raw 32 × 24 frames for one ROI. These small-fixture bounds test queue/data integrity; they are not throughput claims for the CS505MU or any physical rig.

Run the release gates:

python -m pytest --cov=driftless_photometry --cov-report=term-missing
python -m ruff check .
python -m ruff format --check .
python -m build
python -m twine check --strict dist/*

The complete release procedure and one-time Trusted Publisher setup are in the release guide.

See the architecture and the synchronized agent contracts in AGENTS.md and CLAUDE.md. The planned hardware wire contract is documented in the controller protocol. Development priorities, dependencies, and phase exit criteria are maintained in the roadmap.