cellmap-analyze

A suite of Dask-powered tools for processing and analyzing terabyte-scale 3D segmentation datasets. Supports both isotropic and anisotropic voxel sizes.

Features

Processing Tools

These tools chain together into a full predictions-to-segmentations pipeline: start from raw model output (connected-components thresholds prediction probability maps, or mws agglomerates affinities) to get an initial instance segmentation, optionally refine it (split-narrow-bridges to separate accidentally-merged instances, fill-holes, morphological-operations), then finalize with clean-connected-components and/or filter-ids.

Each processing command that writes a volume then adds a multiscale pyramid to it, as multiscale does, so every output is ready to view. Set multiscale: false in its run-config.yaml to skip this; compute-edt adds one only with multiscale: true.

Analysis Tools

Anisotropic data

All operations handle anisotropic voxel sizes (e.g. (8, 8, 32) nm in ZYX). Physical-unit parameters like minimum_volume_nm_3, contact_distance_nm, and gaussian_smoothing_sigma_nm are automatically converted to the appropriate per-axis voxel units. When two datasets have different voxel sizes, they are resampled onto the same voxel grid, using nearest-neighbor interpolation for labels; raw intensities in measure are block-averaged when read at a coarser voxel size.

A roi in a run-config is in nm, ["z_start:z_end", "y_start:y_end", "x_start:x_end"], and selects the voxels whose centres lie inside it; the output covers just those voxels.

Datasets are read through worldarray: zarr v2/v3, N5 and Neuroglancer precomputed, from local paths or s3://, gs:// and http(s)://. Its README documents where each format places voxel 0.

Installation

pip install cellmap-analyze

Usage

All commands share the same basic interface:

<command> [options] <config_path>

• <command>: One of the processing or analysis tools listed above.

• <config_path>: Directory containing:

  • run-config.yaml (parameters for your chosen command)
  • dask-config.yaml (Dask cluster settings)

Options:

• -n, --num-workers N: Number of Dask workers to launch.

Output: A new directory named config_path-<YYYYMMDDHHMMSS> will be created, containing copies of your configs and an output.log for monitoring.

Configuration Examples

The following run-config.yaml could be used to run connected-components.

run-config.yaml

input_path: /path/to/predictions.zarr/mito/s0
output_path: /path/to/segmentations.zarr/mito
intensity_threshold_minimum: 0.71
minimum_volume_nm_3: 1E7
delete_tmp: true
connectivity: 1
mask_config:
  cell:
    path: /path/to/masks.zarr/cell/s0
    mask_type: inclusive
fill_holes: true

dask-config.yaml

The following dask-config.yaml files can be used for a variety of tasks.

Local

jobqueue:
  local:
    ncpus: 1
    processes: 1
    cores: 1
    log-directory: job-logs
    name: dask-worker

distributed:
  scheduler:
    work-stealing: true

LSF Cluster

jobqueue:
  lsf:
    ncpus: 8        # cores per job chunk
    processes: 12  # worker processes per chunk
    cores: 12      # threads per process (1 thread each)
    memory: 120GB  # 15 GB per slot
    walltime: 08:00
    mem: 120000000000
    use-stdin: true
    log-directory: job-logs
    name: cellmap-analyze
    project: charge_group

distributed:
  scheduler:
    work-stealing: true
  admin:
    log-format: '[%(asctime)s] %(levelname)s %(message)s'
    tick:
      interval: 20ms
      limit: 3h

Each worker reads and writes with its share of the job's CPUs, ncpus (or cores) divided by processes, at least 1: the example above gives each worker 1 thread. The job script exports it as WORLDARRAY_THREADS.

Submission

To run on 12 dask workers:

Local run example:

connected-components -n 12 config_path

Cluster submit example (LSF):

bsub -n 4 -P chargegroup connected-components -n 12 config_path

Acknowledgements

The center-finding implementation is taken from funlib.evaluate.