AIND IBL Ephys Alignment Preprocessing

Preprocessing pipeline that prepares SmartSPIM histology and Neuropixels electrophysiology data for the IBL ephys alignment GUI. It registers histology volumes to the Allen Common Coordinate Framework (CCF), converts Neuroglancer probe-track annotations into atlas coordinates, and optionally extracts spike-sorted ephys data into IBL ALF format.

Installation

From PyPI:

pip install aind-ibl-ephys-alignment-preprocessing

From source:

git clone https://github.com/AllenNeuralDynamics/aind-ibl-ephys-alignment-preprocessing.git
cd aind-ibl-ephys-alignment-preprocessing
pip install .

Quick start

The package provides a CLI entry point, aind-ibl-preprocess:

aind-ibl-preprocess \
    --data-root /path/to/data \
    --results-root /path/to/results \
    --neuroglancer neuroglancer.json \
    --manifest manifest.csv

CLI options

Validation

Before running the full pipeline, you can check that all required inputs are present and correctly structured:

aind-ibl-preprocess \
    --data-root /path/to/data \
    --results-root /path/to/results \
    --neuroglancer neuroglancer.json \
    --manifest manifest.csv \
    --validate-only

This checks file existence, manifest structure, reference data availability, disk space (warns below 50 GB, errors below 10 GB), and available RAM (minimum 8 GB).

Regenerating only datapackage.json

If preprocessing has already produced the histology/ephys outputs, you can rewrite just the datapackage metadata:

aind-ibl-preprocess \
    --data-root /path/to/data \
    --results-root /path/to/results \
    --neuroglancer neuroglancer.json \
    --manifest manifest.csv \
    --datapackage-only

This path rediscovers asset metadata, infers successful manifest rows from existing xyz_picks*.json outputs, validates referenced files, and writes a fresh datapackage.json. It does not rerun ANTs transforms, zarr reads, probe track conversion, or ephys extraction.

On Code Ocean, previous /results assets are immutable when attached back to a new capsule. Attach the old preprocessed results asset under /data, then copy it into the new run's /results before rewriting the datapackage:

aind-ibl-preprocess \
    --data-root /data \
    --results-root /results \
    --neuroglancer neuroglancer.json \
    --manifest manifest.csv \
    --datapackage-only \
    --source-results /data/<old_preprocessed_results_asset>

--source-results may point either at the prior results asset root, containing <mouseid>/datapackage.json, or directly at the <mouseid> output directory.

Python API

The package exposes a small public API:

from aind_ibl_ephys_alignment_preprocessing import (
    PipelineConfig,
    run_pipeline,
    run_pipeline_async,
)

config = PipelineConfig(
    data_root="/path/to/data",
    results_root="/path/to/results",
    neuroglancer_file="neuroglancer.json",
    manifest_csv="manifest.csv",
    skip_ephys=False,            # set True to skip ephys extraction
    desired_voxel_size_um=25.0,  # multiscale zarr level selection
    num_parallel_jobs=4,         # parallel workers for ephys
)

# Synchronous
results = run_pipeline(config)

# Asynchronous (structured concurrency with TaskGroups)
import asyncio
results = asyncio.run(run_pipeline_async(config))

Each element of results is a ProcessResult containing the probe ID, recording ID, list of files written, and an optional skip reason.

Input data requirements

Data root layout

All input paths can be absolute or relative to data_root. The expected directory structure is:

data_root/
|
|-- neuroglancer.json
|-- manifest.csv
|
|-- smartspim_lca_template/
|   +-- smartspim_lca_template_25.nii.gz
|
|-- allen_mouse_ccf/
|   +-- average_template/
|       +-- average_template_25.nii.gz
|
|-- allen_mouse_ccf_annotations_lateralized_compact/
|   |-- ccf_2017_annotation_25_lateralized_compact.nrrd
|   +-- ccf_2017_annotation_25_lateralized_unique_vals.npz
|
|-- iblatlas_allenatlas/
|   +-- ...
|
|-- spim_template_to_ccf/
|   |-- syn_0GenericAffine.mat
|   +-- syn_1InverseWarp.nii.gz
|
|-- <smartspim_asset>/
|   |-- image_tile_fusing/
|   |   +-- OMEZarr/
|   |       |-- <registration_channel>.zarr/
|   |       +-- <additional_channels>.zarr/
|   +-- image_atlas_alignment/
|       +-- <registration_channel_stem>/
|           |-- ls_to_template_SyN_0GenericAffine.mat
|           +-- ls_to_template_SyN_1InverseWarp.nii.gz
|
|-- <sorted_recording>/
|   +-- ...  (spike sorting output)
|
+-- **/<probe_file>.json

Each of these is described in detail below.

Manifest CSV

A CSV file describing how histology tracks map onto ephys collections. Each row represents one histology track (or one histology shank) mapped to one ephys collection and ephys shank.

The contract is declared once in types.MANIFEST_COLUMNS and drives both the parser and pre-flight validation, so what a run accepts cannot drift from what validation demands. Optional columns are never required: a manifest written before a column existed stays valid.

Required columns:

Optional columns:

Constraints:

• All rows must have the same mouseid.
• The tuple (mouseid, histology_track_id, histology_shank) must be unique.
• The tuple (recording_id, ephys_collection, histology_shank) must be unique for GUI filename generation.
• The tuple (recording_id, ephys_collection, ephys_shank) must be unique for ephys shank mapping.
• For single-stream multi-shank probes, multiple rows share one ephys_collection and differ in ephys_shank.
• For split quadbase probes, multiple rows can share one logical_probe but use different ephys_collection values; each collection can have ephys_shank=0.

Example:

mouseid,sorted_recording,probe_file,histology_track_id,logical_probe,ephys_collection,histology_shank,ephys_shank
mouse001,2024-06-01_rec_sorted,track_annotations_probeA,A0001,probeA,ProbeA,,
mouse001,2024-06-01_rec_sorted,track_annotations_probeB_shank0,B0001,probeB,ProbeB,0,0
mouse001,2024-06-01_rec_sorted,track_annotations_probeB_shank1,B0001,probeB,ProbeB,1,1
mouse001,2024-06-01_rec_sorted,track_annotations_quad_shank3,Q0001,quad0,ProbeD,3,0

Building a manifest from a mapping

Most of a manifest repeats or follows from two choices: which sorted recording each session uses, and which stream each annotated track was recorded on. aind-ibl-build-manifest takes only those, in a TOML file, and derives the rest. Track names are never parsed, because annotators follow no naming convention.

mouseid = "837772"
annotation = "837772-probe-annotations-CK-2026-09-29.json"  # relative to this file
# registration_asset = "..."                                  # optional, per brain

[[session]]
sorted_recording = "ecephys_837772_2026-06-27_14-08-21_sorted_2026-08-27_18-59-49"
# surface_finding = "..."                                     # optional

[session.tracks]
"732_MD_1" = "46101-1"   # one shank of a split quad-base probe
"733_BLA" = "46810"      # a single-stream probe
"740_PL" = { stream = "45883", shank = 1 }  # one shank of a single-stream multi-shank probe; the first shank is 1

aind-ibl-build-manifest 837772.toml -o manifest_837772.csv [--data-root /data]

A stream is the collection parsed from the sort's postprocessed/ analyzer names. They are listed from a copy mounted under --data-root, or else unsigned from that sort's prefix in aind-open-data. How the columns are derived:

• probe_file is the annotation file's name without its extension.
• Split quad-base probe: a stream <probe>-<N>, when the sort also holds another <probe>-<M>, gets logical_probe = <probe>, histology_shank = N - 1 and ephys_shank = 0.
• Any other stream is its own logical_probe. Its shanks stay blank, or both take the given shank's index.

The build writes nothing if any of these fail, and it reports every failure:

• every stream in each sort has a track, and every assigned stream is in its sort;
• every track is a layer of the annotation file with at least two points;
• no track is assigned twice;
• a split stream has exactly one track, and the tracks sharing any other stream have distinct shanks.

Annotated layers left out of the mapping are printed as notes, so a missed track gets noticed.

Neuroglancer JSON

A Neuroglancer state JSON file that contains image source URIs pointing to the SmartSPIM OME-Zarr volumes. The pipeline extracts the first image source URI to locate the SmartSPIM asset directory and discover the registration channel and any additional channels.

The asset directory must contain:

• image_tile_fusing/OMEZarr/<channel>.zarr/ -- fused OME-Zarr volumes
• image_atlas_alignment/<registration_channel_stem>/ -- ANTs registration outputs: the light-sheet to template affine and inverse warp (or supply a separate registration asset as described below)

Separate registration asset

Set the manifest's optional registration_asset column to the transform directory relative to data_root:

mouseid,sorted_recording,probe_file,histology_track_id,ephys_collection,registration_asset
750108,recording_sorted,probeA,A0001,ProbeA,SmartSPIM_750108_reg/ccf_Ex_639_Em_667

Use one path per brain; conflicting nonblank values fail. An absent or blank column uses the stitched asset's registration.

The directory must contain ls_to_template_SyN_0GenericAffine.mat and ls_to_template_SyN_1InverseWarp.nii.gz: ANTs SyN outputs registering this brain's selected channel to the SmartSPIM LCA template, or to the template named by template_to_ccf_asset (below). The separate template-to-CCF registration is still required.

The final directory name selects the channel after stripping a leading ccf_. For this example, the stitched asset must contain image_tile_fusing/OMEZarr/Ex_639_Em_667.zarr/. Images and metadata come from the stitched asset; the override needs only transforms and their sidecar.

Native-frame registrations need a geometry sidecar beside the transforms. Preprocessing checks ls_to_template_transform_information.json, then transform_information.json. Its moving_domain must describe the actual registration input in LPS millimeters and match the selected volume's native or pipeline-anchored frame. Missing or incompatible domains fail. Without a sidecar, preprocessing assumes the pipeline-anchored frame.

QC: Additional-channel CCF volumes (emit_qc=True) also need ls_to_template_SyN_1Warp.nii.gz in the override directory; without it the QC warp fails rather than falling back to the stitched asset's registration.

Generate the sidecar from ANTs outputs

Use the actual fixed template and moving histology images passed to ANTs, including any preprocessing of their geometry. Transform files alone cannot recover these domains. Headers must be 3D, axis-aligned, and in millimeters.

Run uv sync in the preprocessing library checkout, then save this as generate_registration_sidecar.py:

import sys
from pathlib import Path

import SimpleITK as sitk
from aind_ants_transform_sidecar import dump_package, load_package
from aind_registration_utils.domains import ImageHeader, make_syn_sidecar


def read_header(path):
    reader = sitk.ImageFileReader()
    reader.SetFileName(str(path))
    reader.ReadImageInformation()  # Read geometry without loading voxel data.
    if reader.GetDimension() != 3:
        raise ValueError("Registration inputs must be 3D")
    return ImageHeader(
        origin=reader.GetOrigin(),
        spacing=reader.GetSpacing(),
        direction=reader.GetDirection(),
        size=reader.GetSize(),
    )


registration_dir, fixed_image, moving_image = map(Path, sys.argv[1:])
sidecar = make_syn_sidecar(
    fixed_image_header=read_header(fixed_image),
    moving_image_header=read_header(moving_image),
    affine="ls_to_template_SyN_0GenericAffine.mat",
    warp="ls_to_template_SyN_1Warp.nii.gz",
    inverse_warp="ls_to_template_SyN_1InverseWarp.nii.gz",
)
text = dump_package(sidecar)
load_package(text)  # Validate the serialized package.
destination = registration_dir / "ls_to_template_transform_information.json"
destination.write_text(text + "\n")
print(destination)

Pass the transform directory, fixed image, and moving image in that order:

uv run python generate_registration_sidecar.py \
    /path/to/SmartSPIM_750108_reg/ccf_Ex_639_Em_667 \
    /path/to/fixed_template_used_by_ants.nii.gz \
    /path/to/moving_histology_used_by_ants.nii.gz

The script reads only image headers. Transform filenames are relative to the sidecar; rename raw ANTs outputs to the expected names. The schema requires a forward-warp filename, but normal preprocessing needs only the affine and inverse warp. The helper does not verify transform files.

Generate the JSON beside the transforms before uploading the asset; Code Ocean's /data mounts are read-only.

Separate template-to-CCF registration

A brain registered to a template other than the SmartSPIM LCA template (an iDISCO template, say) needs that template's CCF registration. Set the optional template_to_ccf_asset column to its directory relative to data_root, one path per brain like registration_asset:

mouseid,sorted_recording,probe_file,histology_track_id,ephys_collection,registration_asset,template_to_ccf_asset
750107,recording_sorted,probeA,A0001,ProbeA,idisco_reg/750107,idisco_template_2026-10/template_to_ccf

The directory must contain syn_0GenericAffine.mat and syn_1InverseWarp.nii.gz, ANTs SyN outputs with CCF fixed and the template moving; emit_qc=True also needs syn_1Warp.nii.gz. They replace the pipeline's spim_template_to_ccf_syn_*_25 transforms in the probe and QC chains, and the datapackage refers to them under the template_to_ccf asset, named after the directory's first component under data_root. The individual-to-template sidecar under registration_asset must then name this template as its fixed domain.

Reference volumes

These files are required for atlas registration. Default paths are relative to data_root and can be overridden in PipelineConfig.

Probe annotation files

Per-probe Neuroglancer point annotation files. For each row in the manifest, the pipeline searches data_root for a file matching the glob pattern */<probe_file>.<annotation_format> (e.g., */track_annotations_probeA.json).

These files contain 3D point annotations marking the probe track in the SmartSPIM image space. They are converted through the transform chain:

Neuroglancer pixels --> SPIM (LPS) --> Template --> CCF (LPS/um) --> Bregma (IBL)

Electrophysiology data

Unless --skip-ephys is passed, the pipeline expects spike-sorted data at data_root/<sorted_recording>/ for each unique recording in the manifest. The ephys extraction (delegated to aind-ephys-ibl-gui-conversion) converts this into IBL ALF format (e.g., spikes.times.npy, spikes.clusters.npy, channels.localCoordinates.npy, etc.).

Ephys extraction runs once per unique recording ID (deduplicated across manifest rows).

Output structure

All outputs are written under results_root/<mouseid>/:

results_root/
|-- manifest.csv                              # Copy of input manifest
+-- <mouseid>/
    |-- ccf_space_histology/                  # QC only (emit_qc); GUI never reads it
    |   +-- histology_<channel>.nrrd          # Additional channels in CCF space
    |
    |-- image_space_histology/
    |   |-- histology_registration.nrrd       # Registration channel in image space
    |   |-- histology_registration_pipeline.nrrd
    |   |-- ccf_in_mouse.nrrd                 # CCF template warped to image space
    |   +-- labels_in_mouse.nrrd              # CCF labels warped to image space
    |
    |-- track_data/
    |   |-- spim/<probe_id>.*                 # Track coordinates in SPIM space
    |   |-- template/<probe_id>.*             # Track coordinates in template space
    |   |-- ccf/<probe_id>.*                  # Track coordinates in CCF space
    |   |-- bregma_xyz/<probe_id>.*           # Track coordinates in IBL bregma space
    |   +-- datapackage.json                  # Machine-readable output manifest
    |
    +-- <recording_id>/
        +-- <ephys_collection>/
            |-- xyz_picks.json                # Probe track picks (CCF coordinates)
            |-- xyz_picks_image_space.json    # Probe track picks (image space)
            |-- xyz_picks_shank<N>.json       # Per-shank picks (multi-shank only)
            |-- channels.localCoordinates.npy  # Ephys channel table
            |-- channels.rawInd.npy
            |-- channels.contactId.npy          # Optional; new ephys conversion outputs
            |-- channels.shankInd.npy
            +-- band_corr/                    # Full matrices + row_channels.json

The datapackage.json file describes the mouse, recordings, ephys collections, transform chain, histology, channel tables, and track mappings. It references each ephys directory; individual spike/QC files and band_corr/ contents are loaded by convention rather than enumerated in the JSON. See the datapackage contract for asset descriptions versus runtime location resolution, Code Ocean/desktop examples, shank joins, and validation limits. The model can be loaded back with:

from aind_ibl_ephys_alignment_preprocessing.datapackage import load_datapackage

dp = load_datapackage("/path/to/datapackage.json")

Pipeline overview

The pipeline performs the following steps:

1. Asset discovery -- Parse the Neuroglancer JSON to locate the SmartSPIM OME-Zarr volumes and ANTs registration outputs.
2. Validation -- Check that all required inputs exist and are correctly structured.
3. Histology processing -- Reorient the registration channel to CCF space, export additional channels, and compute inverse transforms (CCF template and labels warped back to image space).
4. Probe processing -- For each manifest row, load the Neuroglancer point annotations and convert them through the transform chain (SPIM -> template -> CCF -> bregma), writing coordinate files at each stage and producing xyz_picks.json files for the alignment GUI.
5. Ephys extraction (optional) -- Convert spike-sorted data into IBL ALF format.
6. Manifest generation -- Write datapackage.json describing explicit references and ephys collection directories; preserve their conventional contents, including QC files not individually listed in the JSON.

Adapting to your own data

If you are not using the AIND SmartSPIM pipeline but want to use this package with your own histology and ephys data, the key requirements are:

1. OME-Zarr volumes -- Your fused histology images must be in OME-Zarr format, discoverable via a Neuroglancer JSON state file.
2. ANTs registration outputs -- You need ANTs SyN registration transforms mapping your histology images to a common template and from that template to the Allen CCF. The pipeline expects standard ANTs output files (*_0GenericAffine.mat, *_1InverseWarp.nii.gz).
3. Reference volumes -- The Allen CCF template, lateralized annotation labels, and the SmartSPIM LCA template at 25 um resolution. These can be overridden in PipelineConfig if your paths differ from the defaults.
4. Probe annotations -- Neuroglancer point annotation JSON files marking probe tracks in histology image space.
5. Spike-sorted data (optional) -- If you want ephys extraction, provide spike-sorted output compatible with aind-ephys-ibl-gui-conversion.

The coordinate convention used throughout is the IBL convention: x = ML (right positive), y = AP (anterior positive), z = DV (dorsal positive), with origin at bregma, in meters.

Development

See CONTRIBUTING.md for developer setup, checks, and repository conventions. Coding agents should also read AGENTS.md.

License

This project is licensed under the MIT License - see the LICENSE file for details.