sollertia-experiment

Provides data acquisition and preprocessing runtimes for Sollertia platform data acquisition systems.

PyPI - Version PyPI - Python Version uv Ruff type-checked: mypy PyPI - License PyPI - Status PyPI - Wheel

Detailed Description

This library is part of the Sollertia AI-assisted scientific data acquisition and processing platform, built on the Ataraxis framework and developed in the Sun (NeuroAI) lab at Cornell University. It is the system-agnostic data acquisition and preprocessing runtime of the platform: it manages the hardware, sessions, and preprocessing pipeline for one or more data acquisition systems and is designed to be extended with new systems over time. The Mesoscope-VR two-photon imaging system is the current reference acquisition system that ships with the library, and it builds on the same session data hierarchy, Virtual Reality task driver, and configuration registry into which new systems plug. The library pairs with sollertia-shared-assets, which supplies the AcquisitionSystems-keyed registry backbone against which new systems register, and feeds the downstream data processing pipeline exposed by sollertia-forgery.

Features

  • Supports Windows, Linux, and macOS, with Linux as the intended platform and the only one that receives extensive testing.
  • System-agnostic acquisition and preprocessing runtime, designed to be extended with new acquisition systems.
  • Mesoscope-VR reference acquisition system combining two-photon imaging with a Unity Virtual Reality behavioral task.
  • A single consolidated sle command line interface for hardware discovery, configuration, acquisition, and data management.
  • Hardware abstraction through binding classes for Zaber motors, machine-vision cameras, and microcontrollers.
  • Session-based data management with distributed, redundant long-term storage.
  • An sle mcp MCP server and two Claude Code marketplace plugins, experiment and mesoscope, for AI-assisted operation.
  • Apache 2.0 License.

Table of Contents

Dependencies

Each acquisition system relies on external software and hardware components that must be installed and configured separately, and for the Mesoscope-VR system they are documented under the Mesoscope-VR Data Acquisition System section.

For users, all other library dependencies are installed automatically by all supported installation methods. For developers, see the Developers section for information on installing additional development dependencies.

Installation

Source

Note, installation from source is highly discouraged for anyone who is not an active project developer.

  1. Download this repository to the local machine using the preferred method, such as git-cloning. Use one of the stable releases that include precompiled binary and source code distribution (sdist) wheels.
  2. If the downloaded distribution is stored as a compressed archive, unpack it using the appropriate decompression tool.
  3. cd to the root directory of the prepared project distribution.
  4. Run pip install . to install the project and its dependencies. Alternatively, if using a distribution with precompiled binaries, use pip install WHEEL_PATH, replacing 'WHEEL_PATH' with the path to the wheel file.

pip

Use the following command to install the library and all of its dependencies via pip: pip install sollertia-experiment

Usage

Data Acquisition Systems

A data acquisition (and runtime control) system can be broadly defined as a collection of hardware and software tools used to conduct training or experiment sessions that acquire scientific data. Each data acquisition system can use one or more machines (PCs) to acquire the data, with this library (sollertia-experiment) typically running on the main data acquisition machine. Additionally, each system can use any number of long-term storage destinations, such as a remote storage Server, a Network-Attached-Storage (NAS) volume, or both, to safely store the data after the acquisition (with redundancy and parity).

In the Sollertia platform, every data acquisition system runs a Unity Virtual Reality task in a linear infinite corridor and is built around the main tool used to acquire its scientific data. For example, the current reference system is the Mesoscope-VR system, which is built around the 2-Photon Random Access Mesoscope (2P-RAM) used to acquire brain activity data. Other systems may center on a different acquisition modality. Due to this inherent specialization, each Sollertia platform data acquisition system is treated as an independent unit that contributes its own bindings, runtime logic, and preprocessing steps, while plugging into the shared platform machinery for everything else.

Extending the Platform

The library is designed so that a new acquisition system can be added by composing the shared machinery rather than rewriting it. The reusable substrate spans this library and sollertia-shared-assets:

  • The acquisition-system configuration registry. This library defines a shared SystemConfiguration base class and an AcquisitionSystems-keyed registry in cross_system/system_configuration.py. Each system defines a concrete subclass (the Mesoscope-VR system contributes MesoscopeSystemConfiguration in mesoscope_vr/system.py) and registers it at import time via register_system_configuration(). Registration is the only system-specific wiring the configuration file lifecycle requires, and resolution, creation, and loading of the on-disk YAML are all shared.
  • The shared session data hierarchy and registries. The 4-level root/project/animal/session hierarchy, the SessionData model, and the record registries (session descriptors keyed by session type, hardware states and experiment configurations keyed by acquisition system) are owned by sollertia-shared-assets. A new system contributes its record schemas to those registries and reuses the rest of the hierarchy unchanged.
  • The hardware-agnostic Virtual Reality task driver. The vr_task driver encapsulates the MQTT contract with the Unity game engine, the editor bridge that drives scene activation and Play Mode, and the cue-sequence-to-trial decomposition. The driver is hardware-agnostic, surfacing typed Virtual Reality task events that any acquisition system dispatches against its own hardware.
  • The interfaces/ registration seams. A new system's MCP tools register themselves. interfaces/mcp_server.py globs *_tools.py directly inside the interfaces/ directory and imports every match, so adding interfaces/{system}_tools.py is the whole edit. The CLI side carries no equivalent discovery. A command group reaches the top-level sle group through two hand-written lines, one import and one add_command() call, inside _register_subcommands() in interfaces/entry_points.py.
  • The shared preprocessing primitives. cross_system/data_preprocessing.py provides log archive assembly, video renaming, surgery record snapshotting, storage push, session directory deletion, and project migration. Each system names its behavior DataLogger after BEHAVIOR_LOGGER_NAME, which is how log assembly locates the log directory, and resolves its own configuration into the system-agnostic StorageDestinations interface on which the shared utilities operate.

Note, the platform does not expose a generic runtime base class, and that is a design decision rather than a missing abstraction. An acquisition system's engine is defined by a physical hardware inventory and by laboratory-local wiring conventions that no base class carries honestly. Each system therefore composes its controller from the shared seams above and scaffolds it from the worked example, settling the hardware-defined decisions with the engineer who owns the rig. MesoscopeVRSystem (in mesoscope_vr/system_controller.py) is currently the only acquisition-system controller, and it is that worked example.

For the seam-by-seam catalog of everything a new acquisition system touches across this library and sollertia-micro-controllers, use the experiment plugin's library-extension skill. The assets plugin's library-extension skill owns the matching sollertia-shared-assets registry half, which is the AcquisitionSystems member, the AcquisitionSystems-keyed dispatch registries, the per-system record dataclasses, and the import-time contract checks. Both skills are required to land a new system. From there, system-design-pipeline orders the build phases, acquisition-system-design covers the configuration and binding layer, and acquisition-system-runtime covers the runtime layer. The microcontroller-interface, zaber-interface, and vr-driver-interface skills cover the individual hardware and Unity coupling seams. See AI-Assisted Development for the skill inventory.

Mesoscope-VR Data Acquisition System

This is the current reference Sollertia platform data acquisition system. The system broadly consists of four major parts:

  1. The 2-Photon Random Access Mesoscope (2P-RAM), assembled by Thor Labs and controlled by ScanImage software. The Mesoscope control and data acquisition are performed by a dedicated computer referred to as the 'ScanImagePC'. This PC is assembled and configured by MBF Bioscience. The only modification carried out during assembly was the configuration of Server Message Block (SMB) protocol access to the root directory used by the ScanImage software to save the Mesoscope data.
  2. The Unity game engine running the Virtual Reality game world used in all experiments to control the task environment and resolve the task logic. The virtual environment runs on the main data acquisition computer referred to as the 'VRPC' and relies on the MQTT communication protocol and the sollertia-virtual-reality Unity package to bidirectionally interface with the virtual task environment.
  3. The microcontroller-powered hardware that allows the animal to bidirectionally interface with various physical components (modules) of the Mesoscope-VR system.
  4. A set of visual and IR-range cameras, used to acquire behavior video data.

Note, the hardware inventory, configuration parameters, and calibration workflows for this system are documented in depth by the mesoscope plugin's mesoscope-vr skill. Use the experiment plugin's system-health-check and acquisition-system-setup skills to verify the dependencies below before running sessions.

Main Dependency

  • Linux operating system. The library also runs on Windows and macOS, but it is intended for Linux, which is the only platform on which it is extensively tested. It has been explicitly written for the Ubuntu 26.04 LTS distribution of the GNU Linux operating system, using the Wayland window system architecture and the kernel that ships with that release.

Software Dependencies

Note, This list only includes external dependencies, which are installed in addition to all dependencies automatically installed from pip / conda as part of library installation. The dependencies below have to be installed and configured on the VRPC before calling runtime commands via the command line interface (CLI) exposed by this library.

  • MQTT broker version 2.1.2. The broker runs on the VRPC, and the VRPC-side runtime reaches it on the default loopback IP (127.0.0.1) and Port (1883). Controlling the Mesoscope from a separate ScanImagePC additionally requires exposing the broker on the local network, as described in MQTT Broker Access.
  • FFMPEG. As a minimum, the version of FFMPEG should support H265 and H264 codecs with hardware acceleration (Nvidia GPU). This library was tested with the version 9.0.1.
  • MvImpactAcquire. This library is tested with version 2.9.2, which is freely distributed. Higher GenTL producer versions likely work too, but they require purchasing a license.
  • Zaber Launcher version 2026.5.15-1.
  • Unity Game Engine version 6000.3.23f1 LTS.

Note, face-camera pose inference is an optional preprocessing step, so the dependency below is required only by projects that choose to run it. FFMPEG, MvImpactAcquire, and Zaber Launcher are required by every Mesoscope-VR session. The MQTT broker is required by window checking and experiment sessions, and the Unity Game Engine by experiment sessions alone.

  • A separate conda environment that provides the sollertia-video-tracking library (slvt) and its DeepLabCut installation. Preprocessing invokes conda run -n <env> slvt infer to analyze the face-camera video, because DeepLabCut cannot share this library's Python environment. The step runs only when the video_tracking.conda_environment and video_tracking.dlc_project_path fields of the Mesoscope-VR system configuration are both set, and leaving either unset disables it and leaves the rest of preprocessing unaffected. When they are set but the environment or the model is unavailable, preprocessing aborts the transfer to long-term storage and retains the local session copy for a manual retry.

Hardware Dependencies

Note, These dependencies only apply to the VRPC. Hardware dependencies for the ScanImagePC are determined and controlled by MBF and ThorLabs. This library benefits from the ScanImagePC being outfitted with a 10-Gigabit network card, but this is not a strict requirement.

  • Nvidia GPU. This library uses GPU hardware acceleration to encode acquired video data. Any Nvidia GPU with hardware encoding chip(s) should work as expected. The library was tested with RTX 4090.
  • A CPU with at least 12, preferably 16, physical cores. This library has been tested with AMD Ryzen 7950X CPU. It is recommended to use CPUs with 'full' cores, instead of those using a mixture of 'efficiency' and 'performance' cores for predictable performance of all library components.
  • A 10-Gigabit capable motherboard or Ethernet adapter, such as X550-T2. Primarily, this is required for the high-quality machine vision cameras used to record videos of the animal's face and body. The 10-Gigabit lines are also used for transferring the data between the PCs used in the data acquisition process and the destination machines used for long-term data storage (see acquired data management section for more details).

System Assembly

The Mesoscope-VR system consists of multiple interdependent components. It is continually refined with minor changes to optimize its performance and facilitate novel experiments and projects. Treat this section as a general system composition guide, but consult the original publication for each project over this section for instructions on building specific system implementations used to acquire the data featured in different publications.

Physical assembly and mounting of all hardware components mentioned in the specific subsections below is discussed in the main Mesoscope-VR assembly section.

Zaber Motors

All brain activity recordings with the Mesoscope require the animal to be head-fixed. To orient head-fixed animals on the Virtual Reality treadmill (running wheel) and promote task performance, the system uses three groups of motors controlled through Zaber motor controllers. The first group, the HeadBar, is used to position the animal's head in Z, Pitch, and Roll axes. Together with the movement axes of the Mesoscope, this allows for a wide range of motions necessary to align the Mesoscope objective with the brain imaging plane. The second group of motors, the LickPort, controls the position of the water delivery port (tube) (and sensor) in X, Y, and Z axes. This is used to ensure all animals have comfortable access to the water delivery tube, regardless of their head position. This motor group also controls the position of the aversive air puff valve, which is mounted directly above the LickPort. The third group of motors, the Wheel, controls the position of the running wheel in the X-axis relative to the head-fixed animal's body and is used to position the animal on the running wheel to promote good running behavior.

The current snapshot of Zaber motor configurations used by the Sollertia platform, alongside the motor parts list and electrical wiring instructions, is available in the Zaber configuration snapshot folder.

Warning! Zaber motors have to be configured correctly to work with this library. To (re)configure the motors to work with the library, apply the setting snapshots from the link above via the Zaber Launcher software. Read the instructions in the 'Applying Zaber Configuration' document for the correct application procedure.

Although highly discouraged, it is also possible to edit the motor settings manually. To configure the motors to work with this library, overwrite the non-volatile User Data of each motor device (controller) with the data expected by the shared cross_system Zaber binding layer:

  1. User Data 0: Device CRC Code. This variable should store the CRC32-XFER checksum of the device's name (user-defined name). During runtime, the library generates the CRC32-XFER checksum of each device's name and compares it against the value stored inside the User Data 0 variable to ensure that each device is configured appropriately to work with the sollertia-experiment library. Note, Use the sle get checksum console command to generate the CRC32-XFER checksum for each device during manual configuration, as it uses the same code as used during runtime and, therefore, guarantees that the checksums match.
  2. User Data 1: Device ShutDown Flag. This variable tracks whether the device was properly shut down during previous runtimes. The library sets this variable to 0 when each device is initialized at runtime startup and back to 1 during proper shutdown. This flag is only checked for devices marked as "unsafe" (see User Data 10 below). For unsafe devices that were not properly shut down, the system prompts for user confirmation before re-homing the motor at the beginning of the next runtime.
  3. User Data 10: Device Unsafe Flag. This variable should be set to 1 for motors that can be positioned in a way that is not safe to home after power cycling (e.g., motors that could collide with other components during homing). When a motor marked as unsafe is not properly shut down (User Data 1 = 0), the system prompts the user for confirmation before proceeding with the homing sequence.
  4. User Data 11: Device Park Position. This variable should be set to the position, in native motor units, where the device should be moved as part of the 'park' command and the shut-down sequence. This is used to position all motors in a way that guarantees they can be safely 'homed' at the beginning of the next runtime. Therefore, each park position has to be selected so that each motor can move to its 'home' sensor without colliding with any other motor simultaneously moving towards its 'home' position. Note, The LickPort uses the 'park' position as the default imaging position. During runtime, it moves to the 'park' position if it has no animal-specific position to use during imaging. Therefore, the park position for the LickPort should always be set so that it cannot harm the animal mounted in the Mesoscope enclosure while moving to the park position from any other position.
  5. User Data 12: Device Maintenance Position. This variable should be set to the position, in native motor units, where the device should be moved as part of the 'maintain' command. Primarily, this position is used during water delivery system calibration and the running-wheel surface maintenance. Typically, this position is calibrated to provide easy access to all hardware components of the system by moving all motors as far away from each other as reasonable.
  6. User Data 13: Device Mount Position. This variable should be set to the position, in native motor units, where the device should be moved as part of the 'mount' command. For the LickPort, this position is usually far away from the animal, which facilitates mounting and unmounting the animal from the rig. For the HeadBar and the Wheel motor groups, this position is used as the default imaging position. Therefore, the HeadBar and the Wheel 'mount' positions should be set so that any (new) animal can be comfortably and safely mounted in the Mesoscope enclosure.

Note, for the binding-class patterns behind the User Data scheme and for safe manual motor positioning, use the experiment plugin's zaber-interface skill.

Behavior Cameras

To record the animal's behavior, the system uses two high-end machine-vision cameras with 2-MegaPixel resolution. The face_camera records the animal's face, while the body_camera records the body of the animal. To interface with the cameras, the system leverages customized ataraxis-video-system bindings.

Specific information about the cameras and related imaging hardware, as well as the snapshot of the configuration parameters used by each camera, is available in the camera configuration snapshot folder.

Note, for low-level camera configuration, use the video plugin's camera-interface and camera-setup skills. For how the cameras integrate with the Mesoscope-VR system, use the mesoscope plugin's mesoscope-vr skill.

MicroControllers

To interface with all other hardware components other than cameras and Zaber motors, the Mesoscope-VR system uses Teensy 4.1 microcontrollers with specialized ataraxis-micro-controller code. Currently, the system uses three isolated microcontroller subsystems: Actor, Sensor, and Encoder.

For instructions on assembling and wiring the electronic components used in each microcontroller system, as well as the code running on each microcontroller, see the microcontroller repository.

Note, for the registry of paired Module and ModuleInterface classes, use the experiment plugin's microcontroller-interface skill. The communication plugin's microcontroller-interface skill covers the underlying base API, and the microcontroller plugin's firmware-module skill covers the firmware side.

Virtual Reality Task Environment (Unity)

The task environment used in all Mesoscope-VR experiments is rendered and controlled by the Unity game engine. To make Unity work with this library, each project-specific Unity task must use the bindings and assets released as part of the sollertia-virtual-reality repository. Follow the instructions from that repository to set up the Unity game engine to interface with this library and to create new virtual task environments.

Note, The Unity Editor must be running with the project open before starting an experiment session. The editor exposes an HTTP MCP Bridge that starts automatically with it, and this library uses that bridge to open the correct scene and to 'arm' (enter Play Mode) and 'disarm' (exit Play Mode) the task automatically. The operator does not open scenes or press the Unity 'play' button manually, and they only confirm that the Virtual Reality display renders correctly during setup. Use the sle get unity command to verify the bridge is reachable before running a session.

Coordination via MQTT

The sollertia-experiment library and Unity coordinate the running task bidirectionally using the MQTT protocol. This communication is used to:

  • Verify scene configuration: At startup, sollertia-experiment requests the active Unity scene name and validates it against the scene name expected from the experiment configuration.
  • Decompose the cue sequence: Unity sends the active VR environment's wall cue sequence to sollertia-experiment, which decomposes it into individual trials. The spatial trial layout (per-trial cue sequences and segment lengths) is sourced from the Virtual Reality task template (defined in sollertia-virtual-reality and persisted as vr_configuration.yaml), not from the experiment configuration. The acquisition-side per-trial parameters (reward size, gas puff duration, etc.) from the experiment configuration are then joined back to each decomposed trial by trial name.
  • Synchronize runtime state: sollertia-experiment sends the per-cycle animal motion delta to Unity and receives one stimulus event per trial, resolving each trial's outcome from it. The event carries the trial name, whether the stimulus was delivered, and whether the animal's own behavior or the guidance fallback produced the outcome.
  • Control task guidance: sollertia-experiment sends task guidance state updates based on animal performance.

Two configuration files are preserved with each Virtual Reality task session's raw data for reproducibility: the experiment configuration (experiment_configuration.yaml, the acquisition-side experiment parameters) and the Virtual Reality task template (vr_configuration.yaml, the corridor cues, VR environment, and trial structures Unity renders). Together they capture both halves of the experiment.

Note, for the MQTT topic contract, the editor bridge, and the cue-sequence decomposition, use the experiment plugin's vr-driver-interface skill. For authoring tasks and scenes on the Unity side, use the unity plugin. Task template authoring is owned by the assets plugin's task-templates skill.

Google Sheets API Integration

This library interacts with the shared Google Sheet files used by the Sollertia platform to track and communicate certain information about the animals that participate in all projects. Currently, this includes two files: the surgery log and the water restriction log. Primarily, this integration is used to ensure that all information about each experiment subject (animal) is stored in the same location (on the long-term storage machine(s)). Additionally, it is used to automate certain data logging tasks.

The boundary between the libraries is as follows: the credential-file plumbing (the platform credentials directory and the slsa configure credentials command) is owned by sollertia-shared-assets, while the actual surgery-log and water-log sheet reading and writing is owned by this library. Each sheet is optional and is only processed when its identifier (surgery_sheet_id, water_log_sheet_id) is set in the Mesoscope-VR system configuration.

Setting up Google Sheets API Access

Skip to the next section if a Google Sheets API service account already exists. Most lab members can safely ignore this section, as service accounts are managed at the platform level rather than by individual lab members.

  1. Log into the Google Cloud Console.
  2. Create a new project.
  3. Navigate to APIs & Services → Library and enable the Google Sheets API for the project.
  4. Under IAM & Admin → Service Accounts, create a service account. This generates a service account ID in the format of service-account@gserviceaccount.com.
  5. Use Actions → Manage Keys and, if a key does not already exist, create a new key and download it in JSON format. This key is then used to access the Google Sheets.
Registering the Service Account Key

The downloaded key must be registered with the Sollertia platform on the VRPC before any runtime can read or write the Google Sheets. Register it using the sollertia-shared-assets CLI:

slsa configure credentials -c google -f /path/to/downloaded_key.json

This copies the key into the platform credentials directory under its canonical name (google_credentials.json). Use slsa get credentials -c google to report the registered path. The assets plugin's working-directory skill owns this registration path.

Critical! If the key is not registered but a Google Sheet identifier is configured for the system, the preprocessing runtime aborts with a FileNotFoundError when it attempts to resolve the credentials. Register the key before running sessions that rely on the surgery or water logs.

Adding Google Sheets Access to the Service Account

To access the surgery log and the water restriction log Google Sheets as part of this library's runtime, create and share these log files with the email of the service account created above. The service account requires Editor access to both files.

Note, This feature requires that both log files are formatted according to the available Sollertia platform templates. Otherwise, the parsing algorithm does not behave as expected, leading to runtime failures. Additionally, both log files have to be pre-filled in advance, as the processing code is not allowed to automatically generate new table (log) rows. Note, Currently, it is advised to pre-fill the data a month in advance. Since most experiments last for at most a month, this usually covers the entire experiment period for any animal.

Note, for the sheet schema contract and for authoring a processor for a new sheet layout, use the experiment plugin's google-sheets-processing skill.

ScanImage PC Assets

As mentioned above, the ScanImagePC is largely assembled and configured by external contractors. However, the PC requires additional assets and configuration post-assembly to make it compatible with sollertia-experiment-managed runtimes.

File System Access

To support the sollertia-experiment runtime, the ScanImagePC's filesystem must be accessible to the VRPC via the Server Message Block version 3 (SMB3) or equivalent protocol. Since the ScanImagePC uses Windows, it is advised to use the SMB3 protocol, as all Windows machines support it natively with minimal configuration. As a minimum, the ScanImagePC must be configured to share the root Mesoscope output directory with the VRPC over the local network. This is required to fetch the data acquired by the Mesoscope during preprocessing and to retrieve the desktop screenshot generated during session setup. The Mesoscope acquisition itself is controlled over MQTT, not through the shared directory (see MATLAB Assets).

MQTT Broker Access

The Mesoscope-VR system controls the Mesoscope by exchanging MQTT messages with the runAcquisition MATLAB function running on the ScanImagePC. The MQTT broker runs on the VRPC and is shared with the Unity Virtual Reality task. Because the ScanImagePC connects to this broker over the local network, the broker must be configured to accept non-loopback connections.

By default, Mosquitto binds only to the loopback interface (127.0.0.1), so a separate ScanImagePC cannot reach it. To expose the broker on the local network, add the following to the VRPC's mosquitto.conf:

listener 1883
allow_anonymous true

The listener 1883 directive binds all network interfaces, including loopback, so the VRPC-side runtime and Unity continue to connect on 127.0.0.1 without changes. The allow_anonymous true directive is required because defining an explicit listener disables anonymous access by default, which both the sollertia-experiment runtime and the MATLAB client require.

Note, anonymous access is acceptable because the Sollertia platform operates on an isolated local network. Deployments on shared or untrusted networks should configure broker authentication instead.

After updating the configuration, restart Mosquitto and ensure the VRPC firewall allows inbound connections on the broker port (1883). Finally, pass the VRPC's network address to the MATLAB function when launching the Mesoscope control interface, for example runAcquisition(hSI, hSICtl, broker="tcp://VRPC-IP:1883").

Default Screenshot Directory

During runtime, the sollertia-experiment library prompts the user to generate a screenshot of the ScanImagePC desktop and place it directly in the root of the network-shared ScanImagePC Mesoscope directory. That root is the directory named by the system configuration's mesoscope_directory field (see above), and the screenshot stays out of its mesoscope_data subdirectory, which the runtime requires to be empty before it prepares the Mesoscope. The screenshot is used to store the information about the red-dot alignment, the acquisition parameters, and the state of the imaging plane at the beginning of each session. The library is statically configured to fetch the screenshot from the shared directory and does not look in any other directories. Therefore, it is advised to reconfigure the default output directory used by the 'Win + PrtSc' command on the ScanImagePC to save the screenshots into the shared Mesoscope output directory.

MATLAB Assets

ScanImage software is written in MATLAB and controls all aspects of Mesoscope data acquisition. Every runtime that images the brain requires the experimenter to manually interface with the ScanImage GUI during Mesoscope preparation. Only the window checking and mesoscope experiment session types command the Mesoscope and therefore require the user to call the runAcquisition MATLAB function on the ScanImagePC. This function carries out multiple runtime-critical tasks, including setting up the acquisition, generating and applying the online motion correction, and servicing the acquisition commands the VRPC issues over MQTT. The function connects to the same MQTT broker as the Unity Virtual Reality task, using a dedicated Mesoscope topic namespace that does not overlap with the Unity topics, and reports command reception and progress back to the VRPC. Lick training and run training sessions never command the Mesoscope over MQTT, so they require neither runAcquisition nor the ScanImage GUI. They still resolve the shared ScanImagePC Mesoscope directory as part of their session data layout.

The runAcquisition function ships with this library under assets/mesoscope_vr. See the Mesoscope-VR ScanImage PC assets guide for instructions on deploying the function to the ScanImagePC and registering it with MATLAB's search path. The function relies on the MariusMotionEstimator and MariusMotionCorrector2 online motion-correction classes, which are provided as part of the ScanImage installation on the ScanImagePC, and on the MATLAB Industrial Communication Toolbox, which provides the mqttclient interface.

Note, for the ScanImagePC and runAcquisition setup, use the mesoscope plugin's mesoscope-vr skill.

Mesoscope-VR Assembly

The Mesoscope-VR assembly mostly consists of two types of components. First, it includes custom components manufactured via 3D-printing or machining (for metalwork). Second, it consists of generic components available from vendors such as ThorLabs, which are altered in workshops to fit the specific requirements of the Mesoscope-VR system. The blueprints and CAD files for all components of the Mesoscope-VR system, including CAD renders of the assembled system, are available in the CAD and blueprint folder.

Acquired Data Structure and Management

The library defines a fixed structure for storing all acquired data which uses a 4-level directory tree hierarchy: root (volume), project, animal, and session. This structure is reused by all acquisition systems, and it is maintained across all long-term storage destinations. After each data acquisition runtime (session), all raw data is stored under the root/project/animal/session/raw_data directory stored on one or more machines mentioned below.

Currently, each Sollertia platform data acquisition system uses a main data acquisition PC and any number of long-term storage destinations (zero, one, or more):

  1. The main data acquisition PC is used to acquire and preprocess the data. For example, the VRPC of the Mesoscope-VR system is the main data acquisition PC for that system. This PC is used to both acquire the data and, critically, to preprocess the data before it is moved to the long-term storage destinations.
  2. Each long-term storage destination is a machine or storage volume to which the preprocessed data is transferred for long-term storage. The Mesoscope-VR system anticipates two common destinations by default. The first is a Server, a high-performance compute server used as the primary long-term storage and analysis destination. The second is a NAS, a Network-Attached-Storage volume used for redundant 'cold' backup storage, typically located in a different physical location to provide data storage redundancy. Neither is required: a system can be configured with any number of destinations under arbitrary names, or with none at all.

Critical! Each configured long-term storage destination is expected to be mounted to the main acquisition PC filesystem (for example, using the Server Message Block 3 (SMB3) protocol). Therefore, each data acquisition system operates on the assumption that all configured storage destination filesystems are used contiguously and can be freely accessed by the main acquisition PC's Operating System.

Note, the library transfers the preprocessed data to every configured long-term storage destination and purges the redundant local copy from the acquisition machine once the transfer succeeds. If no destinations are configured, the data is retained on the acquisition machine and the preprocessing is limited to the on-premises data conversion and aggregation steps.

Root Directory (Volume)

All data acquisition systems and all long-term storage destinations keep ALL Sollertia platform projects under the same root directory. The exact location and name of the root directory on each machine is arbitrary but should generally remain fixed (unchanging) over the entire lifetime of that specific machine. On the main acquisition machine, this local data root is configured with the slsa configure data-root command and resolved at runtime, so it is not part of the system configuration file. The root directories of any long-term storage destinations remain configured through the system configuration.

Project Directory

When a new project is created, a project directory named after the project is created under the root directory of the main data acquisition machine. When the data is moved to the configured long-term storage destinations as part of preprocessing, the project directory is also created on these destinations, if it does not already exist.

The slsa configure project command also creates a configuration subdirectory under the root project directory. This directory stores all supported experiment configurations for the project. For the Mesoscope-VR system, the sle mesoscope run command searches the configuration directory for the .yaml file with the name of the target experiment to load the experiment data.

Animal Directory

When the library is used to acquire data for a new animal, it generates a new animal directory under the root and project directory combination. The directory uses the ID of the animal as its name.

All data acquisition systems also use a persistent_data subdirectory under the root animal directory to store data that is reused between data acquisition sessions, and for the Mesoscope-VR system the directory is created by the system runtime as needed.

Critical! The current Sollertia platform convention stipulates that all animal IDs should be numeric. While some library components do accept strings as inputs, it is expected that all animal IDs only consist of positive integers. Failure to adhere to this naming convention can lead to runtime errors and unexpected behavior of all library components!

Session Directory

Each time the library is used to acquire data, a new session directory is created under the root, project and animal directory combination. The session name is derived from the current UTC timestamp at the time of the session directory creation, accurate to microseconds. Primarily, this naming format was chosen to make all sessions acquired by the same acquisition system have unique and chronologically sortable names. The session name format follows the order of YYYY-MM-DD-HH-MM-SS-US.

Raw Data

All data acquired by this library is stored under the raw_data subdirectory, generated for each session. Overall, an example path to the acquired (raw) data can therefore look like this: /media/Data/Experiments/Template/666/2025-11-11-05-03-23-234123/raw_data/.

Note, This library treats both newly acquired and preprocessed data as raw. This is because preprocessing does not change the content of the data. Instead, preprocessing uses lossless compression to more efficiently package the data for transmission, and the data can at any time be converted back to the original format.

Shared Raw Data

Note, For information about the processed data, see the main data processing library.

After acquisition and preprocessing, the raw_data directory of each acquisition system contains, as a minimum, the following files and subdirectories:

  1. ax_checksum.txt: Stores the xxHash3-128 checksum used to verify data integrity when it is transferred to the long-term storage destination. The checksum is generated before the data leaves the main data acquisition system PC and, therefore, accurately captures the final state of the raw data before it enters storage.
  2. hardware_state.yaml: Stores the snapshot of the dynamically calculated parameters used by the data acquisition system modules during runtime. These parameters are recalculated at the beginning of each data acquisition session and are rounded and stored using the appropriate floating point type (usually fp64) to minimize floating point rounding errors. This file is also used to determine which modules were used during runtime and, consequently, which data can be parsed from the .npz log files generated at runtime (see below).
  3. session_data.yaml: Stores information necessary to maintain the same session data structure across all machines used during data acquisition and long-term storage. This file is used by all Sollertia platform libraries as an entry point for working with a session's data. The file also includes all available information about the identity and purpose of the session and can be used by human experimenters to identify the session. The file also stores the versions of the sollertia-experiment library and Python that were used to acquire the data.
  4. session_descriptor.yaml: Stores session-type-specific information, such as the task parameters and experimenter notes. The contents of the file are different for each session type, although some fields are reused by all sessions. The contents for this file are partially written by the library (automatically) and, partially, by the experimenter (manually, at the end of each session). At the end of each runtime, a copy of the descriptor file is cached inside the persistent_data directory of the animal under a session-type-specific name (e.g., lick_training_descriptor.yaml), replacing any already existing copy. This is used to optionally restore certain runtime configuration parameters between session types that support this functionality.
  5. surgery_metadata.yaml: Stores the data on the surgical intervention(s) performed on the animal that participated in the session. This data is extracted from the surgery log Google Sheet, so the file is only generated when a surgery sheet identifier is configured for the system. For most animals, the contents should be the same across all sessions.
  6. system_configuration.yaml: Stores the configuration parameters of the data acquisition system that generated the session data. This is a snapshot of all dynamically addressable configuration parameters used by the system. When combined with the assembly instructions and the appropriate sollertia-experiment library version, it allows completely replicating the data acquisition system used to acquire the session's data.
  7. behavior_data: Stores compressed .npz log files that contain all non-video behavior data acquired by the system. This includes all messages sent or received by each microcontroller, the timestamps for the frames acquired by each camera, and (if applicable) the main brain activity recording device (e.g.: Mesoscope). These logs also include session metadata, such as trials, task conditions, and system and runtime state transitions. Although the exact content of the behavior data directory can differ between acquisition systems, all systems used in the lab generate some form of non-video behavior data.
  8. camera_data: Stores the behavior videos recorded by video cameras used by the acquisition system. Videos are named using the session name and camera identifier (e.g., {session_name}_face_camera.mp4, {session_name}_body_camera.mp4).
  9. experiment_configuration.yaml: This file is only created for experiment sessions. It stores the acquisition-side configuration of the experiment task performed by the animal during runtime. The contents of the file differ for each data acquisition system, but each system generates a version of this file. The file contains enough information to fully replicate the acquisition-side experiment runtime on the same acquisition system and to process and analyze the acquired data.
  10. vr_configuration.yaml: This file is created for every Virtual Reality task session. It stores a snapshot of the Virtual Reality task template, which carries the linear infinite corridor cues, VR environment, and trial structures that the Unity game engine renders during the session, sourced from the sollertia-virtual-reality task template. Together with experiment_configuration.yaml, it contains the information necessary to fully replicate the Virtual Reality environment used during the experiment.

Note, Mesoscope-VR window checking sessions are an exception to entries 2, 7, and 8 above. They do not start the Mesoscope-VR system controller, which is the only writer of hardware_state.yaml, so that file is never generated for them. Their preprocessing also deletes the stub behavior_data and camera_data directories the acquisition camera stack leaves behind, before the data integrity checksum is computed, so neither directory reaches long-term storage.

Note, the generic read, write, and validate surface for session_descriptor.yaml and hardware_state.yaml is owned by the assets plugin's session-descriptors and session-hardware-state skills, and the Mesoscope-VR field schemas of both files are owned by the mesoscope plugin's mesoscope-vr-session-schema skill.

Shared Temporary Data

The session data hierarchy additionally uses the following temporary marker files and directories which are cleared before the raw data is transmitted to the long-term storage destinations:

  1. nk.bin: This marker is automatically cached to disk by the shared session model as part of creating a new session data hierarchy. Each runtime removes this marker file when it successfully completes its runtime preparation. If this marker exists when the runtime enters the shutdown cycle, this indicates that the runtime encountered a fatal error during startup and had to be terminated early. In this case, the session's data is silently deleted, as uninitialized sessions necessarily do not contain any valid data.
  2. behavior_data_log: All behavior log entries are initially saved as individual .npy files. Each .npy file stores a serialized log message in the form of a uint8 (byte) NumPy array. Since messages are cached to disk as soon as they are received by the DataLogger to minimize data loss in case of emergency shutdowns, the temporary behavior_data_log directory is used to store these messages during runtime. Frequently, the directory accumulates millions of .npy files at runtime, making it challenging for human operators to work with the data. During preprocessing, individual .npy files are grouped by their source (what made the log entry, e.g.: VideoSystem, MicroController, Data Acquisition System, etc.) and are compressed into .npz archives, one for each source. The .npz archives are written in place and the individual .npy files are removed to conserve disk space. The behavior_data_log directory is then renamed to behavior_data.

Mesoscope-VR System Data

The Mesoscope-VR system instantiates a directory hierarchy both on the VRPC and the ScanImagePC. Below is the list of files and directories found on each of these machines.

Raw Data

The Mesoscope-VR system generates the following files and directories, in addition to those discussed in the shared raw data section, on the VRPC:

  1. mesoscope_data: Stores all Mesoscope-acquired data. Since Mesoscope data is only acquired for experiment and window checking sessions, this directory is not created for training session types. During preprocessing, the directory contents are organized to automatically work with the cindra processing library. The directory contains:
    • mesoscope_XXXXXX_XXXXXX.tiff: Recompressed Mesoscope frame stacks using LERC lossless compression.
    • frame_invariant_metadata.json: ScanImage metadata constant across all frames (frame rate, plane/channel count, and ROI information).
    • frame_variant_metadata.npz: Per-frame metadata (frame numbers, timestamps, acquisition triggers).
    • cindra_parameters.json: cindra-compatible configuration used for further data processing.
    • MotionEstimator.me: Reference motion estimator file.
    • fov.roi: Region-of-interest file defining the imaging field.
    • zstack.tiff: A high-resolution z-stack reference image for future alignment with histology data.
  2. zaber_positions.yaml: Stores the snapshot of the positions used by the HeadBar, LickPort, and Wheel Zaber motor groups, taken during the session's data acquisition runtime. All positions are stored in native motor units. This file is created for all session types supported by the Mesoscope-VR system. The snapshot is generated exactly once per session. For lick training, run training, and experiment sessions it is taken inside the system shutdown sequence, and for window checking sessions it is taken immediately after the runtime is marked initialized. Generation is skipped when the Zaber motor groups are not connected, and when the session still carries the nk.bin uninitialized-session marker, so no snapshot exists for a session that failed to start. At that single generation point, a copy of the snapshot is also cached inside the persistent_data directory of the animal, overwriting any existing copy, to support restoring the motors to the same position during the next session.
  3. mesoscope_positions.yaml: Stores the snapshot of the Mesoscope objective position taken at the end of the session's data acquisition. The snapshot covers the physical axes (X, Y, Z, and Roll), the virtual ScanImage axes (Fast Z, Tip, and Tilt), the laser power at the sample, and the red-dot alignment Z position. Note, all fields except the red-dot alignment Z position are queried live from the ScanImage software over MQTT, so the mesoscope control driver must still be connected and idle when the snapshot is generated. The red-dot alignment Z position is the only field the experimenter enters, defaulting to the previous runtime's value, and every value is then floored to three decimal places before the file is written. This file is only created for window checking and experiment sessions. A copy of this snapshot file is also saved to the persistent_data directory of the animal to support restoring the Mesoscope to the same imaging field during the next session.
  4. window_screenshot.png: Stores the screenshot of the ScanImagePC screen. The screenshot should contain the image of the red-dot alignment, the view of the target cell layer, the Mesoscope position information, and the data acquisition parameters. Primarily, the screenshot is used by experimenters to quickly reference the imaging quality from each experiment session. This file is only created for window checking and experiment sessions. A copy of this file is saved to the persistent_data directory of the animal to help the user realign the red-dot to a similar position during the next session.
  5. camera_data eye-tracking predictions: The DeepLabCut .h5 prediction file and its companion pickle files, written beside {session_name}_face_camera.mp4 inside camera_data by the slvt infer subprocess during preprocessing. They are only generated for experiment sessions, and only when the system configuration's video_tracking section sets both conda_environment and dlc_project_path, as an unset value disables the inference. A failed inference or a missing prediction file aborts the transfer to long-term storage and retains the local session copy for a manual retry, with the inference log kept in the system temporary directory as slvt_infer_{session_name}.log.
ScanImagePC

All Mesoscope-VR system data on the ScanImagePC is stored under the user-defined ScanImagePC root directory, which is expected to be mounted to the VRPC via the SMB3 or similar protocol. Under that root directory, the system creates the following directories and files:

  1. mesoscope_data: This directory stores all Mesoscope-acquired data for the currently running session. The runAcquisition MATLAB function configures the ScanImage software to output all data to the mesoscope_data directory, which is shared by all sessions, animals, and projects. This allows using the same static output path for all ScanImage acquisitions.
  2. session-specific mesoscope_data: At the end of each runtime, the Mesoscope-VR system renames the mesoscope_data directory to instead use the session name (id). Then, it generates an empty mesoscope_data directory for the next runtime. This way, all data of each completed session is stored under a unique directory named after that session. This step is crucial for data preprocessing, which identifies the session data directory and pulls it over to the VRPC based on the session name (id).
  3. persistent_data: This directory is created for each unique project and animal combination, similar to the data structure created by sollertia-experiment on the main acquisition system PC. This directory contains the MotionEstimator.me and fov.roi files of the animal's first reference-generating session, which is normally its window checking session. Preprocessing persists the pair the first time a session produces it, and every later imaging session that finds an existing pair, whether window checking or experiment, offers to replace it with the reference generated during that session. These files are typically reused by all following data acquisition sessions to restore the imaging field to the location the persisted reference describes. The full path to the persistent_data directory would typically look like root/project/animal/persistent_data.
Mesoscope-VR Temporary Data

The Mesoscope-VR system also generates the following temporary directory during runtime:

  1. raw_mesoscope_frames: Stores uncompressed .TIFF stacks fetched by the VRPC from the ScanImagePC. This is done as part of data preprocessing to collect all data on the VRPC before executing individual preprocessing subroutines. The .TIFF stacks are then re-compressed using the Limited Error Raster Compression (LERC) scheme and are saved to the mesoscope_data directory. Once this process completes successfully, the raw_mesoscope_frames directory with all raw files is deleted to conserve disk space.

Note, the VRPC controls the Mesoscope acquisition over MQTT. See the ScanImage PC Assets section for details on the MQTT command interface.

Acquiring Data

All user-facing library functionality is realized through a set of Command Line Interface (CLI) commands automatically exposed when the library is pip-installed into a python environment. The library exposes a single top-level sle command with two subcommand groups. The first is a general, hardware-agnostic discovery group (sle get) shared by all acquisition systems, and the second is the Mesoscope-VR system group (sle mesoscope) that combines configuration, maintenance, data acquisition, and data management for the Mesoscope-VR system. A standalone sle mcp command starts the MCP server (see AI-Assisted Development). Each group contains subcommands that allow further configuring their runtime.

To use any of the commands described below, activate the python environment where the library is installed, e.g., with conda activate MYENV and type one of the commands described below.

Warning! Most commands described below use the terminal to communicate important runtime information to the user or request user feedback. Carefully read every message printed to the terminal during runtime. Failure to do so may damage the equipment or harm the animal!

CLI Commands

Use the --help argument when calling any of the commands described below to see the list of supported arguments together with their descriptions and default values.

Command Description
sle get Discover and evaluate data acquisition system components (general)
sle mesoscope configure system Generate the Mesoscope-VR data acquisition system configuration file
sle mesoscope configure experiment Create a Mesoscope-VR experiment configuration from a task template
sle mesoscope maintain Run the Mesoscope-VR system maintenance session
sle mesoscope check-bridge Check whether the ScanImagePC runAcquisition loop is reachable
sle mesoscope check-mounts Verify every filesystem path the Mesoscope-VR configuration declares
sle mesoscope validate-config Validate the Mesoscope-VR system configuration and report its issues
sle mesoscope run Execute Mesoscope-VR data acquisition and training sessions
sle mesoscope preprocess Preprocess a session's data and push it to long-term storage
sle mesoscope delete Remove a session's data from all storage destinations
sle mesoscope migrate Transfer an animal's sessions between projects
sle mcp Start the MCP server backing the sle get and sle mesoscope tools

Step 0: Configuring the Data Acquisition System

Before acquiring data, each acquisition system has to be configured. This step is done in addition to assembling the system and installing the required hardware components. Typically, this only needs to be done when the acquisition system configuration or hardware changes, so most lab members can safely skip this step.

Use the sle mesoscope configure system command to generate the system configuration file. The command requires the Sollertia platform working directory to already be set on the host machine, and it writes the configuration file into that directory's configuration folder. Once the write succeeds, the command removes any other acquisition system's *_system_configuration.yaml file from that folder, so the machine belongs to exactly one acquisition system. All future sollertia-experiment runtimes resolve that single *_system_configuration.yaml file from the working directory automatically, and fail when the folder does not contain exactly one such file.

Note, the Sollertia platform working directory that receives the configuration file is a platform-level setting shared across libraries, so it is configured separately from the system configuration file. Set it on the main acquisition machine with the slsa configure directory command (from the sollertia-shared-assets library) and view it with the slsa get directory command. The sle mesoscope configure system command aborts with a FileNotFoundError when the working directory has not been set, when its cached path record is empty, or when the configured directory no longer exists.

Note, the local data root is configured separately from the system configuration file, because it is a Sollertia platform-level setting shared across libraries. It is the directory under which all projects, animals, and sessions are stored on the main acquisition machine. Set it on the main acquisition machine with the slsa configure data-root command (from the sollertia-shared-assets library) and view it with the slsa get data-root command. The data root must be configured before running data acquisition sessions. The root directories of any long-term storage destinations remain part of the system configuration. The assets plugin's working-directory skill owns the data root, the working directory, and the platform credentials.

Note, Each acquisition system uses unique configuration parameters. Additionally, the sollertia-experiment library always assumes that any machine (PC) can only be used by a single data-acquisition system (is permanently a part of that acquisition system). Only the main PC of the data acquisition system (e.g.: the VRPC of the Mesoscope-VR system) that runs the sollertia-experiment library should be configured via this command.

The system configuration parameters are defined and owned by this library (sollertia-experiment): the SystemConfiguration base class lives in cross_system/system_configuration.py, and the Mesoscope-VR system's concrete MesoscopeSystemConfiguration lives in mesoscope_vr/system.py. For information about the available system configuration parameters, read the API documentation of this library and the mesoscope plugin's mesoscope-vr skill. Use the experiment plugin's system-health-check skill to validate the configuration before running sessions.

Step 1: Creating a Project

All data acquisition sessions require a valid project to run. To create a new project, use the slsa configure project command (from the sollertia-shared-assets library). This command can only be called on the main PC of a properly configured data-acquisition system (see Step 0 above). As part of its runtime, this command generates the project directory, together with its configuration subdirectory, under the local data root of the main acquisition PC only. The matching project directories on the configured long-term storage destinations are created later, when preprocessing transfers a session's data to them (see the Project Directory section above). Use the assets plugin's project-hierarchy skill for project creation and for walking the project tree.

Step 2: Creating an Experiment

All projects that involve scientific experiments also need to define at least one experiment configuration. Experiment configurations are unique for each data acquisition system and are stored inside .yaml files named after the experiment. To generate a new experiment configuration file, use the sle mesoscope configure experiment command. This command generates a precursor experiment configuration file inside the configuration subdirectory, stored under the root project directory on the main PC of the data acquisition system. The command takes three required options and five optional ones:

Option Description
-p, --project The project that receives the new experiment configuration file (required)
-e, --experiment The name given to the experiment and to its configuration file (required)
-t, --template The task template to instantiate, as the filename without the .yaml extension (required)
-sc, --state-count The number of runtime states the experiment supports (default 1)
--reward-size The default water reward volume for lick-type trials, in microliters (default 5.0)
--reward-tone-duration The default reward tone duration for lick-type trials, in milliseconds (default 300)
--puff-duration The default gas puff duration for occupancy-type trials, in milliseconds (default 100)
-f, --force Overwrites an existing experiment configuration file (default off)

Note, the command resolves the template named by -t, --template through the host machine's configured task templates directory, which is a platform-level setting owned by sollertia-shared-assets rather than part of the system configuration file. Point the main acquisition PC at the sollertia-virtual-reality project's Assets/InfiniteCorridorTask/Configurations directory with the slsa configure templates -d /path/to/Assets/InfiniteCorridorTask/Configurations command before running this command, and view the configured path with the slsa get templates command. The command aborts with a FileNotFoundError when that directory has not been configured, when its cached path record is empty, or when the configured directory no longer exists.

For information about the available experiment configuration parameters in the precursor file, use the mesoscope plugin's mesoscope-vr-experiment-schema skill, which owns the field schema, the trial classes, and the from_task_template seeding this command performs. The task template named by -t, --template is owned by the assets plugin's task-templates skill.

Mesoscope-VR Note: Mesoscope-VR experiments also require a valid Virtual Reality environment generated through the sollertia-virtual-reality package. The experiment configuration supplies the acquisition-side, per-trial parameters, while the Virtual Reality task template defines the corridor cues, VR environment, and spatial trial structures that Unity renders (persisted with the session as vr_configuration.yaml). The two are joined by trial name at runtime. After creating the experiment configuration, use the Unity package to generate the corresponding VR scene before running experiment sessions.

Step 3: Discovering System Components

Before running sessions, use sle get subcommands to discover and verify the components accessible to the data acquisition system:

Command Description
sle get zaber Discovers Zaber motor devices connected to the system
sle get cameras Discovers cameras accessible to the system (OpenCV and Harvesters interfaces)
sle get controllers Discovers microcontrollers accessible to the system
sle get ports Lists available serial communication ports
sle get unity Checks whether the Unity Editor MCP Bridge is reachable
sle get checksum -i STRING Calculates the CRC32-XFER checksum for the input string

Note, project and experiment discovery is provided by the sollertia-shared-assets library. Use slsa get projects and slsa get experiments -p PROJECT to list the projects and experiment configurations stored under the data root.

Note, the Mesoscope-VR pre-flight checks live in the sle mesoscope group rather than in the sle get group, because each one reads the Mesoscope-VR system configuration. Use sle mesoscope check-mounts to verify every filesystem path the configuration declares, sle mesoscope validate-config to report every issue the configuration carries, and sle mesoscope check-bridge to confirm the ScanImagePC's runAcquisition control loop is reachable before starting a Mesoscope imaging session.

Note, for an orchestrated pre-flight verification of configuration, storage mounts, and hardware connectivity, use the experiment plugin's system-health-check and acquisition-system-setup skills.

Step 4: Maintaining the Acquisition System

All acquisition systems contain modules that require frequent maintenance. Most of these modules are unique to each acquisition system.

Mesoscope-VR

The Mesoscope-VR system contains three modules that require frequent maintenance: the water delivery system, the gas puff valve, and the running wheel. To facilitate the maintenance of these modules, the sollertia-experiment library exposes the sle mesoscope maintain command.

This command is typically called at least twice during each day the system is used to acquire data. First, it is used at the beginning of the day to prepare the Mesoscope-VR system for runtime by filling the water delivery system. Second, it is used at the end of each day to empty the water delivery system.

Less frequently, this command is used to re-calibrate the water delivery system, typically as a result of replacing system components, such as tubing or the valve itself. The command is also occasionally used to replace the surface material of the running wheel, which slowly deteriorates as the wheel is used.

This command can also facilitate cleaning the wheel, which is typically done before and after each runtime to remove any biological contaminants left by each animal participating in experiment or training runtimes.

The maintenance GUI's Gas Puff Valve Control section opens and closes the gas puff valve and delivers single puffs of a configurable duration, from 10 to 350 milliseconds, so the operator can purge and test the air line.

Note, This runtime uses a GUI-based interface for issuing commands and monitoring the system state.

Step 5: Acquiring Data

Each acquisition system supports one or more distinct types of data-acquisition sessions (runtimes). The set of supported session types is unique to each system, and commonly this includes an 'experiment' session type, which is the primary use case for most acquisition systems in the lab. Some systems may also support one or more training session types, which often do not acquire any brain activity data, but otherwise behave similarly to experiment sessions.

Session commands take common parameters: user ID, project name, animal ID, and animal weight. These are provided to the parent sle mesoscope run command before specifying the session type. Each session type requires only the parameters it records, so the window checking session runs without the animal weight.

Mesoscope-VR Session Commands

The Mesoscope-VR system supports four types of runtime sessions:

1. Window Checking Session

sle mesoscope run -u USER -p PROJECT -a ANIMAL window-checking

This session guides the user through finding the imaging plane and generating the reference MotionEstimator.me, fov.roi, and zstack.tiff files for the checked animal. This session is typically used 2 to 3 weeks after the surgical intervention and before any training or experiment sessions to assess the quality of the intervention and the suitability of including the animal in experiment cohorts.

2. Lick Training Session

sle mesoscope run -u USER -p PROJECT -a ANIMAL -w WEIGHT lick-training [OPTIONS]

All animals that participate in Mesoscope-VR experiments undergo a two-stage training protocol, with lick training being the first stage. During this runtime, the animals are head fixed in the Mesoscope enclosure for ~20 minutes. The primary goal of the runtime is to teach the animals to lick at the water tube to consume water rewards and associate the sound tone emitted at reward delivery with water coming out of the tube. During runtime, the running wheel is locked, so the animals cannot run.

Optional arguments include maximum training time, reward delay parameters, maximum water volume, and unconsumed reward tracking threshold.

3. Run Training Session

sle mesoscope run -u USER -p PROJECT -a ANIMAL -w WEIGHT run-training [OPTIONS]

This is the second stage of the mandatory two-stage Mesoscope-VR training protocol. During this runtime, the animals are head fixed in the Mesoscope enclosure for ~40 minutes. The primary goal of the runtime is to teach the animals to run on the wheel while head-fixed and to associate running with receiving water rewards. During runtime, the running wheel is unlocked, but the Virtual Reality screens are kept off, ensuring that the animal is not exposed to any visual cues until the first experiment day.

Optional arguments include maximum training time, speed and duration thresholds, threshold increase parameters, maximum water volume, idle time allowance, and unconsumed reward tracking threshold.

4. Experiment Session

sle mesoscope run -u USER -p PROJECT -a ANIMAL -w WEIGHT experiment -e EXPERIMENT [OPTIONS]

This session type is designed to execute the experiment specified in the target experiment_configuration.yaml file (see above). The system supports varying experiment configurations and Virtual Reality environments.

The -e/--experiment argument specifies the name of the experiment configuration to use.

Note, for the runtime state machine, the session types, and the behavior visualizer / control UI behind these commands, use the mesoscope plugin's mesoscope-vr-runtime skill.

Step 6: Preprocessing and Managing Data

All acquisition systems support several ways of handling the session's data acquired at runtime. For most runtimes, the choice of how to handle the data is made as part of the acquisition system shutdown sequence. However, in the case of unexpected runtime terminations, all data preprocessing steps can also be executed manually by calling the appropriate CLI command.

Note, the preprocess, delete, and migrate operations below are also exposed as sle mcp tools, and the experiment plugin's data-management skill drives them (including bulk preprocessing) for AI-assisted use.

Preprocessing Data

The most commonly used operation is to preprocess the acquired data. This can be done manually by calling:

sle mesoscope preprocess -sp SESSION_PATH

Preprocessing consists of two major steps. The first step pulls all available data to the main data acquisition system machine (PC) and re-packages (re-compresses) the data to reduce its size without loss. The second step distributes (pushes) the data to all configured long-term storage destinations, such as a Server and a NAS.

Note, a session that still carries the nk.bin uninitialized-session marker never finished runtime initialization. Preprocessing such a session purges it from the host machine, every configured long-term storage destination, and the ScanImagePC instead of preprocessing it. The purge runs without a confirmation prompt, which is the same silent-delete rule the nk.bin marker entry above applies to the runtime shutdown cycle.

Critical! It is imperative that all valid data acquired in the lab undergoes preprocessing as soon as possible. Only preprocessed data is stored in a way that maximizes its safety by using both redundancy and parity. Data that is not preprocessed may be lost in the case of emergency, which is considerably less likely for the preprocessed data.

When preprocessed data is successfully and safely pushed to long-term storage destinations, the preprocessing runtime removes the local copy of the session's data from the data acquisition system machine to conserve disk space.

Deleting Session Data

The second way of managing the data is primarily used during testing and when handling interrupted sessions that did not generate any valid data. This involves removing all session data from both the data acquisition system and all long-term storage destinations. This runtime is extremely dangerous and, if not used carefully, can permanently delete valid data. This mode can be triggered using:

sle mesoscope delete -sp SESSION_PATH

Warning! This command is not recommended for most users. The CLI form carries no command-line confirmation gate, so it reaches the purge as soon as it is invoked, while the delete_session_tool MCP tool refuses to act until the caller passes confirm_deletion. The purge itself still opens a blocking terminal confirmation prompt for any session that finished initialization, and skips that prompt only for a session that still carries the nk.bin uninitialized-session marker.

Migrating Animal Data Between Projects

To transfer all sessions for an animal from one project to another, use:

sle mesoscope migrate -s SOURCE_PROJECT -d DESTINATION_PROJECT -a ANIMAL_ID

This moves the animal's data across all accessible storage destinations: the sessions are relocated to the destination project, not copied.

Recovering from Interruptions

While it is not typical for the data acquisition or preprocessing pipelines to fail during runtime, it is possible. The library can recover or gracefully terminate the runtime for most code-generated errors, so this is usually not a concern. However, if a major interruption (e.g., power outage) occurs or one of the hardware assets malfunctions during runtime, manual intervention is typically required to recover the session's data and reset the acquisition system.

Data acquisition interruption

Data acquisition can be interrupted in two main ways, the first being due to an external asset failure, for example, if the ScanImagePC unexpectedly shuts down during Mesoscope-VR system runtime. In this case, the runtime pauses and instructs the user to troubleshoot the issue and then resume the runtime. This type of soft interruption is handled gracefully during runtime to exclude the data collected during the interruption from the output dataset. Generally, soft interruptions are supported for most external assets, which include anything not managed directly by the sollertia-experiment library and the main data acquisition system PC. While inconvenient, these interruptions do not typically require specialized handling other than recovering and restoring the failed asset.

Note, While most soft interruptions typically entail resuming the interrupted runtime, it is also possible to instead terminate the runtime. To do so, press the Terminate Runtime button in the runtime GUI instead of trying to resume the runtime. In this case, the system attempts to execute a graceful shutdown procedure, saving all valid data in the process.

The second way involves interruption due to sollertia-experiment runtime failures or unexpected shut-downs of the main acquisition system PC. In these cases, manual user intervention is typically required to recover the useful data and reset the system before the acquisition can be restarted. The handling of such cases often consists of specific steps for each supported acquisition system. Typically, these hard interruptions are related to major issues, such as global facility power loss or severe malfunction of sensitive acquisition system components, such as microcontrollers and communication cables.

Mesoscope-VR

If the VRPC runtime unexpectedly interrupts at any point without executing the graceful shutdown, follow these instructions:

  1. If the session involved Mesoscope imaging, shut down the Mesoscope acquisition process and make sure all required files (frame stacks, MotionEstimator.me, fov.roi, and zstack.tiff) have been generated and saved to the mesoscope_data directory.
  2. If necessary, manually edit the session_descriptor.yaml, the mesoscope_positions.yaml, and the zaber_positions.yaml files to include actual runtime information. Estimate the volume of water delivered at runtime by manually reading the water tank level gauge.
  3. Remove the animal from the Mesoscope enclosure. If necessary, use the Zaber Launcher app to directly interface with Zaber motors and move them in a way that allows the animal to be recovered from the enclosure.
  4. Use Zaber Launcher to manually move the HeadBar group's Roll axis to have a positive angle (> 0 degrees). This is critical! If this is not done, the motor cannot home during the next session and instead collides with the movement guard, at best damaging the motor and, at worst, the Mesoscope.
  5. If the session involved Mesoscope imaging, rename the mesoscope_data directory to use the session name. For example, from mesoscope_data → 2025-11-11-05-03-23-234123. Critical! The mesoscope_data directory is shared by every session, so the next runtime clears it and deletes any leftover Mesoscope files it holds. Preprocessing performs the same rename itself, but only while no later runtime has written to the shared directory.
  6. Call sle mesoscope preprocess -sp SESSION_PATH and provide the path to the session directory of the interrupted session. This preprocesses and transfers all collected data to the long-term storage destinations. This preserves any data acquired before the interruption and prepares the system for running the next session.

Note, for inspecting or patching the per-session Zaber and Mesoscope position snapshots edited in step 2, use the mesoscope plugin's mesoscope-vr-snapshots skill (it owns the snapshot write tools). For the runtime recovery behavior, use the mesoscope-vr-runtime skill, and for safe manual motor positioning, the experiment plugin's zaber-interface skill.

Data preprocessing interruption

To recover from an error encountered during preprocessing, call sle mesoscope preprocess -sp SESSION_PATH and provide the path to the session directory of the interrupted session. Re-running the command repeats the whole pipeline rather than resuming from a checkpoint, and each stage aborts early once its input has already been consumed. When the re-run stops with an error naming missing files, perform the cleanup that message requests before retrying.

API Documentation

See the API documentation for the detailed description of the library's internal modules and the CLI commands with their arguments. The top-level sollertia_experiment package does not export any Python-level API symbols, and all user-facing functionality is realized through the sle command line interface.

AI-Assisted Development

The library is built for AI-assisted operation. It ships a single MCP server, exposed through the sle mcp command, and a set of Claude Code skills distributed through the sollertia marketplace. Human operators interact with the library through the sle CLI documented above, and AI agents work through the MCP server tools and the skills described here.

MCP Server

This library provides a single MCP server that exposes two tool sets for AI agent integration, the hardware-agnostic tools of interfaces/get_tools.py and the Mesoscope-VR tools of interfaces/mesoscope_vr_tools.py. The server intentionally does not re-expose assets owned by the sollertia-shared-assets, ataraxis-video-system, and ataraxis-communication-interface dependencies, which are available through the dependencies' own MCP servers.

Starting the Server

Start the MCP server using the CLI:

sle mcp

The -t/--transport option selects the transport, either stdio (the default) or streamable-http.

Available Tools

The seven hardware-agnostic tools and the six sle get commands overlap on three pairs only, get_zaber_devices_tool with sle get zaber, get_checksum_tool with sle get checksum, and check_unity_bridge_tool with sle get unity. The camera, microcontroller, and serial port discovery commands have no tool, and the four tools that read, write, and validate Zaber non-volatile settings or probe a filesystem mount have no CLI command.

Tool Description
get_zaber_devices_tool Discovers Zaber motor devices connected to the system
get_checksum_tool Calculates the CRC32-XFER checksum for an input string
get_zaber_device_settings_tool Reads configuration from a Zaber device's non-volatile memory
set_zaber_device_setting_tool Writes configuration to a Zaber device's non-volatile memory
validate_zaber_configuration_tool Validates a Zaber device's configuration for binding library use
check_mount_accessibility_tool Verifies a filesystem path exists and is writable
check_unity_bridge_tool Checks whether the Unity Editor MCP Bridge is reachable

The Mesoscope-VR tools cover a different surface from the sle mesoscope CLI layer. Six commands pair one to one with a tool, check-bridge, check-mounts, validate-config, preprocess, delete, and migrate, and configure system shares the configuration write path with write_system_configuration_tool. The other six commands, configure experiment, maintain, and the four run subcommands, have no tool, and the eight configuration inspection and session snapshot tools have no CLI command.

Tool Description
read_system_configuration_tool Loads the Mesoscope-VR system configuration
write_system_configuration_tool Creates or replaces the Mesoscope-VR system configuration
validate_system_configuration_tool Validates the system configuration and reports mount status
verify_camera_configuration_tool Compares each camera's live GenICam config to its .yaml
describe_system_configuration_schema_tool Returns the recursive Mesoscope-VR system configuration schema
check_system_mounts_tool Verifies all configured filesystem paths are accessible
check_mesoscope_bridge_tool Checks whether the ScanImagePC runAcquisition loop is reachable
read_session_zaber_positions_tool Loads a session's ZaberPositions snapshot
write_session_zaber_positions_tool Creates or replaces a session's ZaberPositions snapshot
read_session_mesoscope_positions_tool Loads a session's MesoscopePositions snapshot
write_session_mesoscope_positions_tool Creates or replaces a session's MesoscopePositions snapshot
read_session_system_configuration_tool Loads a session's system configuration snapshot
preprocess_session_tool Preprocesses a session's data on the host machine
delete_session_tool Removes a session from all storage locations
migrate_animal_tool Transfers all sessions for an animal between projects

Warning! set_zaber_device_setting_tool and delete_session_tool refuse to act without an explicit confirmation argument. Calling either one without confirm or confirm_deletion returns an error that previews the pending change or spells out the consequences, so the calling agent asks the user before retrying with yes or no.

Client Registration

MCP server registration and the system-agnostic Claude Code skill assets for this library are distributed through the sollertia marketplace as part of the experiment plugin, which is the only plugin that registers the sle mcp server. The Mesoscope-VR system-specific skills ship separately, in the mesoscope plugin. Install the experiment plugin from the marketplace to register the sle mcp server with compatible clients and make the core skills available, and install the mesoscope plugin alongside it to add the six Mesoscope-VR skills listed below.

Marketplace Plugins

The sollertia marketplace ships five plugins, three of which declare an MCP server of their own. Installing a plugin makes its skills available and, when that plugin declares a server, registers that server with compatible clients:

Plugin Targets MCP server Role
assets sollertia-shared-assets slsa mcp Configuration authoring and shared session/subject/template asset I/O
unity sollertia-virtual-reality (none) Unity task authoring, VR scenes, and the MQTT contract, via the slsa relay
experiment sollertia-experiment, sollertia-micro-controllers sle mcp System-agnostic core: design, runtime, hardware interfaces, data management
mesoscope sollertia-experiment, sollertia-shared-assets, sollertia-forgery (none) Mesoscope-VR system-specific skills (layered on the core plugins)
forging sollertia-forgery slf mcp Downstream behavior processing and analysis

The unity plugin uses the Unity relay served by the assets plugin's slsa MCP server and the McpBridge editor plugin, and it requires the assets plugin. The mesoscope plugin requires the experiment plugin's sle mcp server and the assets plugin's slsa mcp server, and it requires the forging plugin's slf mcp server for the processing and forging half. Low-level hardware work additionally draws on the ataraxis marketplace's video and communication plugins and their MCP servers. The sle mcp server intentionally omits the assets, video, and communication tools, which are served by those dependencies' own MCP servers.

Skills

The experiment plugin ships the system-agnostic core skills, and the mesoscope plugin ships the Mesoscope-VR system-specific skills. The table below lists the skills that target this library. The mesoscope plugin also ships seven downstream processing skills that target the sollertia-forgery library instead. With the exception of system-health-check, which is user-invocable as a slash command, the skills are orchestrated by AI agents rather than invoked directly by operators.

Plugin Skill Purpose
experiment pipeline Orchestrate the end-to-end experiment lifecycle of an existing system
experiment system-design-pipeline Orchestrate designing and building a new acquisition system
experiment acquisition-system-design Design pattern: system configuration, binding classes, runtime orchestrator
experiment acquisition-system-runtime Runtime pattern: per-mode logic, state machine, dispatch, control UI
experiment acquisition-system-setup Discover and verify connected acquisition hardware
experiment library-extension Extension seams for a new acquisition system across sle and slmc
experiment system-health-check Pre-flight checks of configuration, mounts, and hardware (user-invocable)
experiment zaber-interface Implement Zaber motor interfaces and binding classes
experiment microcontroller-interface Paired Module + ModuleInterface registry and conventions
experiment vr-driver-interface VR task driver, Unity MQTT contract, and cue-sequence trial decomposition
experiment google-sheets-processing Implement Google Sheets SurgeryLog / WaterLog processors
experiment data-management Preprocess, migrate, and delete session data via sle mcp
experiment external-tool-bindings Bind a tool that cannot be installed beside the stack (producer seam)
experiment cli-reference Document the sle root, sle mcp, and sle get commands and options
experiment experiment-mcp-environment-setup Diagnose sle mcp server connectivity issues
mesoscope mesoscope-vr Mesoscope-VR hardware inventory, configuration, and bindings
mesoscope mesoscope-vr-runtime Mesoscope-VR state machine, orchestrator, UIs, and sle mesoscope CLI
mesoscope mesoscope-vr-snapshots Read/write per-session Zaber and Mesoscope position snapshots
mesoscope mesoscope-vr-session-schema Mesoscope-VR session descriptor and hardware-state field schema
mesoscope mesoscope-vr-experiment-schema Mesoscope-VR experiment configuration and trial-class field schema
mesoscope mesoscope-vr-cli-reference Reference for the sle mesoscope commands, options, and MCP tool mapping

Development assets for this repository ship separately, through the ataraxis marketplace's automation plugin, which carries the Claude Code skills and other AI development assets shared by all Sollertia platform repositories. Install that plugin from the marketplace to make all associated development skills and tools available to compatible AI coding agents.

Developers

This section provides installation, dependency, and build-system instructions for the developers that want to modify the source code of this library.

Installing the Project

Note, this installation method requires mamba version 2.3.2 or above. Currently, all automation pipelines require that mamba is installed through the miniforge3 installer.

  1. Download this repository to the local machine using the preferred method, such as git-cloning.
  2. If the downloaded distribution is stored as a compressed archive, unpack it using the appropriate decompression tool.
  3. cd to the root directory of the prepared project distribution.
  4. Install the core development dependencies into the base mamba environment via the mamba install tox uv tox-uv command.
  5. Use the tox -e create command to create the project-specific development environment followed by tox -e install command to install the project into that environment as a library.

Additional Dependencies

In addition to installing the project and all user dependencies, install the following dependencies:

  1. Python version 3.14. This library currently supports a single Python version (requires-python >=3.14,<3.15). It is recommended to use a tool like pyenv to install and manage the required version.

Development Automation

This project uses tox for development automation. The following tox environments are available:

Environment Description
lint Runs ruff formatting, ruff linting, and mypy type checking
stubs Generates the py.typed marker and .pyi stub files using the project's sdist distribution
docs Builds the API documentation via Sphinx
build Builds sdist and wheel distributions
upload Uploads distributions to PyPI via twine
deploy Uploads the built documentation to the Netlify site
install Builds and installs the project into its sle_dev mamba environment
uninstall Uninstalls the project from its sle_dev mamba environment
create Creates the project's sle_dev mamba development environment (Python 3.14)
remove Removes the project's sle_dev mamba development environment
provision Removes and recreates the sle_dev mamba environment
export Exports the mamba environment as a .yml file
import Creates or updates the mamba environment from the stored .yml file

Run any environment using tox -e ENVIRONMENT. For example, tox -e lint.

Note, all pull requests for this project have to successfully complete the tox task before being merged. To expedite the task's runtime, use the tox --parallel command to run some tasks in parallel.

Automation Troubleshooting

Many packages used in tox automation pipelines (uv, mypy, ruff) and tox itself may experience runtime failures. In most cases, this is related to their caching behavior. If an unintelligible error is encountered with any of the automation components, deleting the corresponding cache directories (.tox, .ruff_cache, .mypy_cache, etc.) manually or via a CLI command typically resolves the issue.

Versioning

This project uses semantic versioning. See the tags on this repository for the available project releases.

Authors

License

This project is licensed under the Apache 2.0 License: see the LICENSE file for details.

Acknowledgments

  • All Sun (NeuroAI) lab members for providing the inspiration and comments during the development of this library.
  • The creators of all other dependencies and projects listed in the pyproject.toml file.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (5.0.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for Inkaros from gravatar.com Inkaros

Credits

Author: Ivan Kondratyev, Kushaan Gupta, Natalie Yeung, Katlynn Ryu, Jasmine Si

Maintainer: Ivan Kondratyev

License expression

Apache-2.0
View SPDX License List

Requires

Python <3.15, >=3.14

Tags

ataraxis data-acquisition experiment mesoscope sollertia

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sollertia_experiment-5.0.1.tar.gz (315.2 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

sollertia_experiment-5.0.1-py3-none-any.whl (278.1 kB view details)

Uploaded Python 3

File details

Details for the file sollertia_experiment-5.0.1.tar.gz.

File metadata

  • Download URL: sollertia_experiment-5.0.1.tar.gz
  • Upload date:
  • Size: 315.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/7.0.0 CPython/3.14.6

File hashes

Hashes for sollertia_experiment-5.0.1.tar.gz
Algorithm Hash digest
SHA256 4b80de1660c15630979d4ca8e7fe64efa903926cae9f542adb4083b1e1031b80
MD5 4530b93b19ae7f0d3058180eb2d3f00a
BLAKE2b-256 83832b3d0bc071e3da582cb76bd88a11c497025252d2e949f08bda2a3020c794

See more details on using hashes here.

File details

Details for the file sollertia_experiment-5.0.1-py3-none-any.whl.

File metadata

File hashes

Hashes for sollertia_experiment-5.0.1-py3-none-any.whl
Algorithm Hash digest
SHA256 f8597c1339dfbc207773d8ac423f7038618db8b0ec9ae1733a8c8c6f263343a2
MD5 9e5bab09e505c6cab552f62b7749dab1
BLAKE2b-256 60b8a80926ecade11340dc443659676e021f5e22613148acb649beed4c501519

See more details on using hashes here.

Release history Release notifications | RSS feed

5.0.2

2 files

This release

5.0.1 This release

2 files

5.0.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page