2P-proc (optimouse)

2P-proc (optimouse) is a 2-photon calcium imaging data processing pipeline built on CaImAn, Mesmerize and Suite2p.

This package provides:

• Motion correction: Rigid and non-rigid motion correction with optional z-drift correction (rigid or non-rigid)
• ROI extraction: CNMF-based or Suite2p-based cell detection and segmentation
• Spike deconvolution: Temporal activity inference from calcium signals
• Parameter optimization: Notebooks for systematic parameter exploration (CaImAn NoRMCorre and CNMF)
• Multi-session registration: Cross-session ROI alignment using CaImAn's registration algorithm

No-installation Quick Start

Try the pipeline without installation, using containers:

• Create your config JSON as described in the Quick Start section
• Edit scripts/2P_proc.sh to set paths and options:

# Copy and customize the batch script
cp scripts/2P_proc_template.sh scripts/2P_proc.sh

• Run the pipeline

# On a local machine
bash scripts/2P_proc.sh configs/my_config.json
# On a cluster with SLURM
sbatch scripts/2P_proc.sh configs/my_config.json

The Docker containers are available on Docker Hub. If using Docker, they will be pulled automatically when running the batch script. Singularity images can be built from the Docker containers using the provided scripts in containers/2P_proc/.

Installation

If you prefer to install the pipeline locally, follow the instructions below.

Standard Installation

# Create and activate conda environment
conda create -n toupie python=3.10 -c conda-forge # Name the env any name you prefer
conda activate toupie
# Install optimouse from PyPI
pip install optimouse
# Install CaImAn manager
caimanmanager install

For Developers

# Clone repository
git clone https://github.com/vncntprvst/2P-proc.git
cd 2P-proc

# Create and activate environment
conda create -n toupie python=3.10 -c conda-forge # Name the env any name you prefer
conda activate toupie
# Install in editable mode
pip install -e .
# Optional: install visualization
pip install -e ".[viz]"
# Install CaImAn manager
caimanmanager install

Additional dependencies

For visualization:

pip install "fastplotlib[notebook]"
pip install plotly

Quick Start

1. Configure your pipeline

Create a configuration JSON based on pipeline/configs/config_template_cnmf.json or pipeline/configs/config_template_suite2p.json.

CNMF-based extraction (config_template_cnmf.json):

{
  "experimenter": "FirstName LastName",
  "subject": {
    "name": "SUBJECT_ID",
    "sex": "M/F/U",
    "genotype": "Strain or Genotype"
  },
  "imaging": {
    "date": "SESSION_DATE",
    "fr": 20,
    "Npixel_x": 765,
    "Npixel_y": 765,
    "microns_per_pixel": 1.45
  },
  "paths": {
    "data_paths": [
      "DATA_ROOT/SUBJECT_ID/SESSION_DATE/TSeries-001"
    ],
    "export_paths": [
      "EXPORT_ROOT/SUBJECT_ID/SESSION_DATE/cnmf"
    ],
    "zstack_paths": [
      "DATA_ROOT/SUBJECT_ID/SESSION_DATE/ZSeries-004"
    ]
  },
  "params_mcorr": {
    "method": "caiman",
    "main": {
      "strides": [36, 36],
      "overlaps": [24, 24],
      "max_shifts": [12, 12],
      "pw_rigid": true
    },
    "save_mcorr_movie": "tiff",
    "z_motion_correction": {
      "zstack_shift": {
        "Ch": 2,
        "Nz": 41
      },
      "non_rigid": false
    }
  },
  "params_extraction": {
    "method": "cnmf",
    "main": {
      "p": 0,
      "K": 8,
      "gSig": [5, 5],
      "merge_thr": 0.8,
      "min_SNR": 3.0,
      "rval_thr": 0.85,
      "use_cnn": true
    }
  },
  "logging": {
    "log_path": "EXPORT_ROOT/SUBJECT_ID",
    "log_level": "INFO"
  }
}

Suite2p-based extraction (config_template_suite2p.json):

{
  "experimenter": "FirstName LastName",
  "subject": {
    "name": "SUBJECT_ID",
    "sex": "M/F/U"
  },
  "imaging": {
    "date": "SESSION_DATE",
    "fr": 20,
    "Npixel_x": 765,
    "Npixel_y": 765
  },
  "paths": {
    "data_paths": [
      "DATA_ROOT/SUBJECT_ID/SESSION_DATE/TSeries-001"
    ],
    "export_paths": [
      "EXPORT_ROOT/SUBJECT_ID/SESSION_DATE/suite2p"
    ],
    "zstack_paths": [
      "DATA_ROOT/SUBJECT_ID/SESSION_DATE/ZSeries-004"
    ]
  },
  "params_mcorr": {
    "method": "caiman",
    "main": {
      "strides": [36, 36],
      "overlaps": [24, 24],
      "max_shifts": [12, 12],
      "pw_rigid": true
    },
    "save_mcorr_movie": "tiff"
  },
  "params_extraction": {
    "method": "suite2p",
    "main": {
      "decay_time": 0.3
    }
  }
}

Notes:

• params_extraction.main accepts generic Suite2p ops overrides (passed through at ops creation).
• See notes on parameters, and setting up a local Suite2p environment for testing: suite2p/readme.md.
• decay_time is mapped to Suite2p tau. Use decay_time or tau interchangeably.

2. Run the pipeline

On a local machine:

conda activate toupie # Activate your conda environment
python -m pipeline.pipeline_mcorr configs/my_config.json
python -m pipeline.pipeline_cnmf configs/my_config.json

On a cluster with SLURM:

Copy and customize scripts/2P_proc_template.sh to scripts/2P_proc.sh, then submit:

sbatch scripts/2P_proc.sh configs/my_config.json

3. Multi-session registration

Match cells across separately processed sessions of the same field of view (Suite2p or CNMF exports):

python -m pipeline.register_sessions --exports <export_day1> <export_day2> ... --out <dir> --method roicat

roicat (default) and track2p run in an environment with that package (--python, or $ROICAT_PYTHON / $TRACK2P_PYTHON); ROICaT needs Python >= 3.11. caiman runs in the pipeline environment. Outputs: cell_registration.npz/.mat (cells x sessions ROI indices in extraction order; 1-based in the .mat), cell_registration_qc.json and cell_registration.png. Method comparison and validation: validation/registration/. Runs recorded the same day are better processed jointly (concatenated) than registered.

Pipeline Components

Motion Correction

• Rigid and non-rigid motion correction using NoRMCorre algorithm
• Optional rigid or non-rigid z-drift correction using anatomical z-stack
• Export to multiple formats (TIFF, memmap, HDF5)

ROI Extraction

• [Option 1] CNMF (CaImAn): Constrained non-negative matrix factorization with automatic component quality assessment
• [Option 2] Suite2p: Fast, correlation-based clustering and SVD-accelerated cell detection and extraction Both methods provide denoised fluorescence traces and deconvolved spikes

ROI Z-Correction

• Regress and subtract axial drift from ROI traces
• Compatible with both CaImAn and Suite2p outputs
• Preserves temporal dynamics while correcting for z-motion

Parameter Optimization

Jupyter notebooks in Mesmerize/ for systematic parameter exploration:

• optimize_mcorr_bruker.ipynb: Motion correction parameters
• optimize_cnmf_bruker.ipynb: CNMF extraction parameters
• pipeline_notebook_template.ipynb: Complete pipeline template (Needs to be updated)

Testing

Unit tests and the z-motion benchmark run on a small real-anatomy fixture (a downsampled subset of C57_O1M2/10022023: one short TSeries + its ZSeries reference stack), published as a GitHub release asset.

# Download the test fixture (~9 MB) into Mesmerize/test_data
./scripts/download_test_data.sh

# Fast unit tests + synthetic z-motion benchmark (no data download needed)
pytest tests/

# Proper z-motion benchmark on real anatomy: impose a known 3D trajectory on
# the real z-stack and check compute_zcorrel recovers it, then run a
# qualitative pass on the actual recording.
python tests/benchmarks/benchmark_zcorr.py \
    --real-zstack Mesmerize/test_data/ZSeries-10022023-1300-004 \
    --real-movie  Mesmerize/test_data/TSeries-10022023-1300-001 \
    --xy-amplitude 1 --deform-amp 0.5 --save-results results/

The fixture is regenerated from raw Bruker data with scripts/make_test_fixture.py (kept as small as possible so it can live as a release asset).

Container Support

Docker/Singularity containers are available for cluster computing:

• containers/2p_proc: Main processing container with optimouse installed
• containers/suite2p: Suite2p-based extraction pipeline
• containers/allenneuraldynamics: Integration with AIND extraction capsule

Build and use containers:

# Build container
cd containers/2p_proc
./build.sh # For Docker only
./build_docker_singularity.sh # For Singularity too

# Run with Singularity
singularity exec containers/2p_proc_latest.sif python -m pipeline.pipeline_mcorr configs/my_config.json

Releasing

Pushing a version tag publishes optimouse to PyPI (.github/workflows/release.yml, PyPI trusted publishing):

# after bumping version in pyproject.toml and committing
git tag v0.1.6 && git push origin v0.1.6

The tag must match the version in pyproject.toml. The 2p_proc container installs the version pinned in containers/2p_proc/Dockerfile; bump that pin and the image tag in build.sh / build_docker_singularity.sh, then rebuild.

Documentation

• Configuration: See pipeline/configs/config_template_cnmf.json and pipeline/configs/config_template_suite2p.json for available parameters
• Tutorials: Jupyter notebooks in Mesmerize/ and Caiman/

Post-Processing and Analysis

For post-processing analysis (behavior integration, population analysis, etc.), see the companion repository: Analysis 2P - Matlab-based analysis pipeline with DeepLabCut and Rastermap integration

Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes with tests
4. Submit a pull request

Citation

This pipeline builds upon the following tools:

• CaImAn: Giovannucci et al. (2019) eLife
• Mesmerize: Kolar et al. (2019) Nature Communications
• Suite2p: Pachitariu et al. (2017) bioRxiv

License

This project is licensed under the MIT License - see LICENSE.md for details.

Support

• Issues: GitHub Issues