aind-ecephys-s3-to-phy

Convert AIND ecephys sorted recordings from S3 into phy directories for manual curation, with raw trace access via compressed binary (mtscomp).

Two S3 assets are needed per session: a sorted asset (postprocessed SortingAnalyzer Zarrs) and its corresponding raw asset (compressed Zarr recordings). The tool reconstructs the full SpikeInterface preprocessing chain — including bad-channel removal — from the analyzer, streams traces from S3, and compresses them to .cbin for phy's trace viewer.

Installation

pip install aind-ecephys-s3-to-phy

Quick start

SORTED=s3://aind-open-data/ecephys_786867_2025-09-25_12-43-56_sorted_2025-12-24_08-46-14
RAW=s3://aind-open-data/ecephys_786867_2025-09-25_12-43-56

# List available recordings (and matched raw files)
s3-to-phy list --sorted-uri $SORTED --raw-uri $RAW

# Convert all recordings to phy (streams from S3, compresses to .cbin)
s3-to-phy convert --sorted-uri $SORTED --raw-uri $RAW

# Convert only recordings for a specific probe
s3-to-phy convert --sorted-uri $SORTED --raw-uri $RAW --probe 46101

# Convert only a specific experiment
s3-to-phy convert --sorted-uri $SORTED --raw-uri $RAW --experiment experiment1

# Write no traces: phy reads them from the raw asset on S3 when you view them
s3-to-phy convert --sorted-uri $SORTED --raw-uri $RAW --probe 46101 --lazy

# Extract preprocessed traces as flat binary (for non-phy pipelines, e.g. mipmap)
s3-to-phy extract-recording \
  --analyzer-uri $SORTED/postprocessed/experiment1_...46101_recording1.zarr \
  --raw-uri $RAW/ecephys/ecephys_compressed/experiment1_...46101.zarr \
  --output preprocessed.dat

# Compress an existing recording.dat to .cbin
s3-to-phy compress --phy-dir ./phy_output

Motion correction

convert and extract-recording correct the traces with the drift estimate Kilosort 4 applied, read through aind-ephys-assets from the sort's spikesorted/motion/<recording>/. Sorts made before sorting capsule cdef667 (2026-04-16), the example asset above among them, did not save it; those export uncorrected, and preprocessing's never-applied dredge_fast estimate is not used as a substitute. Integer traces stay integer. Pass --no-motion-correct to skip it.

Lazy traces

--lazy exports in about a minute per probe instead of hours, with no trace file:

• recording.json holds the analyzer's preprocessing chain over the raw Zarr on S3, and params.py's dat_path names it. phylib opens it through SpikeInterface: this needs a phylib with SpikeInterfaceEphysReader (cortex-lab/phylib#68).
• Waveforms come from the analyzer's stored subset, written as _phy_spikes_subset.* (most of the folder's size, a few GB per probe). Phy reads traces only in the trace view, or for a waveform outside the subset (about 0.5 s per spike from S3). No PC features are computed.
• Traces are read on demand: a jump to a new window costs about a second, and scrolling is cached.
• Traces are not motion-corrected, matching the analyzer's waveforms. SpikeInterface's motion interpolation does not survive a round trip through JSON.

Python API

from aind_ecephys_s3_to_phy.analyzer import load_analyzer, attach_recording, discover_pairs
from aind_ecephys_s3_to_phy.recording import reconstruct_recording
from aind_ecephys_s3_to_phy.export import export_phy

# Discover and convert
pairs = discover_pairs(sorted_uri, raw_uri)
analyzer_name, raw_name = pairs[0]

analyzer_zarr = f"{sorted_uri}/postprocessed/{analyzer_name}"
raw_zarr = f"{raw_uri}/ecephys/ecephys_compressed/{raw_name}"

analyzer = load_analyzer(analyzer_zarr)
recording = reconstruct_recording(analyzer_zarr, raw_zarr)
attach_recording(analyzer, recording)
export_phy(analyzer, recording, "./phy_output", compress=True)

# Or just get the preprocessed recording for other pipelines
recording = reconstruct_recording(analyzer_zarr, raw_zarr)
# → SpikeInterface BaseRecording (filtered, bad channels removed, 373 ch)

S3 credentials

Public buckets (like aind-open-data) work without credentials. For private buckets, configure AWS credentials via the standard methods (~/.aws/credentials, environment variables, IAM roles). SpikeInterface uses s3fs/fsspec which follows the standard AWS credential chain.

Development

uv sync                                # Set up environment
./scripts/run_linters_and_checks.sh -c  # Full lint + test suite
uv run --frozen pytest -m integration   # Integration tests (requires S3)

License

This project is licensed under the MIT License - see the LICENSE file for details.