Skip to main content

zarr-particle-tools

zarr-particle-tools provides a full subtomogram averaging (STA) pipeline and standalone jobs for subtomogram extraction, reconstruction, CTF refinement, and Bayesian polishing. Tilt series can be read from local or S3-backed MRC files and OME-Zarr, including datasets in the CryoET Data Portal.

Extraction and reconstruction are Python implementations of the corresponding RELION 5 jobs and stream Zarr pixels without first downloading the complete tilt series. CTF refinement and polishing run RELION against Zarr through temporary MRC files, preferring /dev/shm and falling back to the system temp directory. zarr-particle-pipeline connects the jobs through py2rely for SLURM-based STA workflows.

Contents

Quickstart (container)

relion-zarr-sta-client installs py2rely and zarr-particle-tools natively (needed for sbatch, which requires the host's own SLURM setup) and wires them to run RELION and the job commands below through the relion-zarr-sta container instead of a local RELION build. On a SLURM cluster, this is enough to run the full pipeline:

apptainer pull relion-zarr-sta.sif oras://ghcr.io/czimaginginstitute/relion-zarr-sta-sif:5.0-cuda12.8
pip install git+https://github.com/czimaginginstitute/relion-docker.git#subdirectory=client
relion-zarr-sta-client --sif relion-zarr-sta.sif --out ~/relion-shims/bin --wire-py2rely
export PATH="$HOME/relion-shims/bin:$PATH"

--wire-py2rely points py2rely's SLURM job scripts at the container automatically; the PATH export lets you also run any of the commands below directly. See relion-docker's client/ for details. Otherwise, see Installation below for a standalone/manual setup.

Installation

conda create -n zarr-particle-tools python=3.12
conda activate zarr-particle-tools
pip install uv
uv pip install zarr-particle-tools
  • Extraction and reconstruction need no RELION installation.
  • CTF refinement and polishing require the corresponding relion_tomo_* binaries.
  • The full pipeline has the additional prerequisites below.

The [pipeliner] extra installs CCPEM pipeliner for the four registered job wrappers:

uv pip install "zarr-particle-tools[pipeliner]"

Installing py2rely in the same environment also installs ccpem-pipeliner.

Pipeline prerequisites

The Quickstart container already has all of the below; skip this section if you're using it.

zarr-particle-pipeline requires:

  • py2rely on PATH.
  • RELION 5 binaries on PATH. Source RELION's setup-env.sh or add its build/bin directory.
  • This package installed in the same environment as py2rely so its ccpem_pipeliner.jobs entry points are registered.
  • A SLURM cluster with sbatch available.

Run zarr-particle-pipeline preflight to check py2rely, RELION, ccpem-pipeliner, and job registration. SLURM availability is not checked. The same preflight runs automatically before pipeline preparation.

Commands

Every command takes --help. Sources are named consistently: local (your own star files), copick-local, data-portal, copick-data-portal (copick picks + portal tilt series; copick run names must be portal run IDs).

Command Purpose Sources
zarr-particle-pipeline Full STA pipeline via py2rely local, copick-local, data-portal, copick-data-portal, plus preflight
zarr-particle-extract Subtomogram extraction (2D stacks) local, copick-local, data-portal, copick-data-portal
zarr-particle-reconstruct Particle map reconstruction local, copick-local, data-portal, copick-data-portal
zarr-particle-ctfrefine RELION relion_tomo_refine_ctf on zarr local, data-portal, copick-data-portal
zarr-particle-polish RELION relion_tomo_align on zarr local, data-portal, copick-data-portal
zarr-particle-tomograms Write a tomograms.star data-portal, copick-data-portal
zarr-particle-importtomo Import portal runs into a RELION project (pipeliner: zarrparticletools.importtomo) data-portal (IDs or a stored selection)
zarr-particle-export Self-contained on-disk project (downloads tilt series) data-portal, copick-data-portal

core/ can also be used directly for projection matrices and point projection, CTF premultiplication, dose weighting, Fourier cropping, masking, backprojection, interpolation, symmetry, and Zarr/S3 I/O.

Usage

Full pipeline

zarr-particle-pipeline:

  • Resolves the selected tilt series and particles.
  • Derives the pixel size and cross-checks portal metadata against each tilt-series MRC header.
  • Writes star files to <output-dir>/input.
  • Runs py2rely prepare relion5-parameters and py2rely prepare relion5-pipeline, then submits pipeline.sh unless --prepare-only is set.

The generated tomograms.star carries a tomoTiltSeriesURI column. py2rely uses that column to select the Zarr extraction, reconstruction, CTF-refinement, and polishing jobs. Refine3D, Class3D, MaskCreate, and PostProcess continue to use stock RELION jobs.

The following portal example uses dataset 10426 and the cytosolic ribosome picks from deposition 10358: 5,246 oriented picks across 38 tilt series at 2.165 Å/px, with per-tilt alignment and CTF metadata.

export TEMPLATE=tests/templates/ribo80s_emd_3883_866_64_resized.mrc
export OUT=10426_ribosome_sta

zarr-particle-pipeline data-portal \
  --dataset-ids 10426 \
  --deposition-ids 10358 \
  --annotation-names "cytosolic ribosome" \
  --output-dir "$OUT" \
  --reference-template "$TEMPLATE" \
  --protein-diameter 330 \
  --box-scaling 1.65 \
  --num-gpus 4 \
  --nthreads 4 \
  --prepare-only
  • --protein-diameter is required.
  • --reference-template is required unless --run-denovo-generation is set.
  • --run-ids limits the selection to specific runs.
  • The portal form also accepts deposition, dataset, organism, run, tilt-series, alignment, tomogram, and annotation filters by ID or name.
  • --pixel-size overrides the derived pixel size; --pixel-size-tol adjusts the MRC-header check.
  • --timeout is the per-job limit in hours; --num-days sets the SLURM walltime request.
  • --run-class3d --nclasses N --class-selection auto enables 3D classification.
  • --gpu-constraint accepts one SLURM GPU feature or a comma- or pipe-separated set interpreted as OR. py2rely drops requested features that are unavailable on the cluster.
  • --prepare-only writes all_sta_parameters.json and pipeline.sh without submitting. Submit later with cd "$OUT" && sbatch pipeline.sh.

Run zarr-particle-pipeline data-portal --help for the complete option list and current defaults.

Star files are written under $OUT/input. RELION job directories such as Extract/, Refine3D/, Class3D/, CtfRefine/, Polish/, and PostProcess/ are created alongside them as the pipeline runs. End-to-end runs can take days on SLURM.

The repository includes tests/templates/ribo80s_emd_3883_866_64_resized.mrc, an 80S ribosome map (EMD-3883) resampled to 8.66 Å/px in a box of 64 (330 × 1.65 / 8.66 = 62.9, rounded up). The path above assumes a source checkout; otherwise set TEMPLATE to your own reference. Match its voxel size to the dataset's coarsest requested binning.

When the picks live in copick instead, copick-data-portal takes them from there and the tilt series from the portal (the copick run names must be portal run IDs):

zarr-particle-pipeline copick-data-portal \
  --copick-config config.json \
  --copick-name ribosome --copick-user-id picker --copick-session-id 1 \
  --copick-dataset-ids 10426 \
  --output-dir ribosome_sta \
  --reference-template template.mrc \
  --protein-diameter 330 \
  --box-scaling 1.65 \
  --num-gpus 4 --nthreads 4

The science options are the same as for data-portal.

For existing star files, both files must be inside --output-dir because py2rely resolves their paths relative to the project directory. The pixel size is read from rlnTomoTiltSeriesPixelSize. Zarr jobs are selected only when tomograms.star contains tomoTiltSeriesURI; otherwise py2rely uses stock RELION jobs.

zarr-particle-pipeline local \
  --output-dir my_sta \
  --particles-starfile my_sta/input/particles.star \
  --tomograms-starfile my_sta/input/tomograms.star \
  --protein-diameter 330 \
  --reference-template ref.mrc

copick-local builds particles.star from copick picks and uses the supplied tomograms.star for optics. It writes the particles file to <output-dir>/input:

zarr-particle-pipeline copick-local \
  --output-dir my_sta \
  --tomograms-starfile my_sta/input/tomograms.star \
  --copick-config pick.json \
  --copick-name ribosome --copick-user-id user0 --copick-session-id 1 \
  --protein-diameter 330 \
  --reference-template ref.mrc

Extraction

For RELION projects, --tiltseries-relative-dir can be omitted if you run from the project root.

zarr-particle-extract local \
  --particles-starfile tests/data/relion_project_synthetic/particles.star \
  --tomograms-starfile tests/data/relion_project_synthetic/tomograms.star \
  --tiltseries-relative-dir tests/data/relion_project_synthetic/ \
  --output-dir tests/output/sample_local_test/ \
  --box-size 16 --bin 4

Extract a larger box and crop it back down with --crop-size (which must be even and no larger than --box-size), --no-ctf to skip CTF premultiplication, and --no-circle-crop for noisier real data:

zarr-particle-extract local \
  --particles-starfile tests/data/relion_project_unroofing/particles.star \
  --tomograms-starfile tests/data/relion_project_unroofing/tomograms.star \
  --tiltseries-relative-dir tests/data/relion_project_unroofing/ \
  --output-dir tests/output/sample_local_test/ \
  --box-size 64 --bin 1 --no-ctf --no-circle-crop

Polished RELION-5 projects can carry per-tilt 2D image deformations (tomo Bayesian polishing with "model 2D deformations": rlnTomoDeformationGridSizeX/Y and rlnTomoDeformationType in tomograms.star, rlnTomoDeformationCoefficients in each tilt-series star). They are applied to every projected particle position exactly as relion_tomo_subtomo does (spline, linear and Fourier models), on top of any --trajectories-starfile motion; pass --ignore-deformations to extract without them. Skipping them on a project that was refined with them mis-centres the crops by the local field value, typically tens of Å. zarr-particle-reconstruct local / copick-local extract through the same step, so they apply deformations too, matching relion_tomo_reconstruct_particle, and take the same flag.

From the portal without first materializing the tilt series locally:

zarr-particle-extract data-portal \
  --run-id "16848, 16851" \
  --annotation-names "ribosome" \
  --ground-truth --inexact-match \
  --output-dir tests/output/sample_data_portal_test/ \
  --box-size 128 --bin 2

Reconstruction

zarr-particle-reconstruct local \
  --particles-starfile tests/data/relion_project_unroofing/reconstruct_particles.star \
  --tomograms-starfile tests/data/relion_project_unroofing/tomograms.star \
  --tiltseries-relative-dir tests/data/relion_project_unroofing/ \
  --output-dir tests/output/sample_local_reconstruct_test/ \
  --box-size 384 --crop-size 256

Reconstruction applies 2D deformations as relion_tomo_reconstruct_particle does.

CTF refinement and polishing

  • CTF refinement runs relion_tomo_refine_ctf; polishing runs relion_tomo_align.
  • Zarr tilt series are streamed into temporary MRC files for RELION to read, in a job-owned directory under --shm-dir (default /dev/shm) that is removed when the job ends, fails or receives SIGTERM (--keep-shm keeps it). When --shm-dir is unusable the system temp directory is used, unless --require-ram-staging is given: then the job fails before streaming unless --shm-dir is tmpfs with room for the staged stacks. This also covers the temporary copies of S3-hosted MRC stacks. The pipeliner jobs require RAM staging by default (do_require_ram_staging).
  • --relion-args passes additional arguments to every RELION call (the pipeliner jobs' other_args).
  • Both jobs require a refined particles.star and reference half-maps from a prior Refine3D job.
  • There is no copick-local variant because raw copick picks are not refined particles.
  • --per-tomogram is the default and uses a staging-bounded two-phase workflow. --all-at-once stages every selected tilt series together.
  • --n-workers 0 selects the worker count automatically.
zarr-particle-ctfrefine local \
  --particles-starfile refined_particles.star \
  --tomograms-starfile tomograms.star \
  --ref1 half1.mrc --ref2 half2.mrc \
  --box-size 384 --do-defocus --do-scale \
  --output-dir tests/output/sample_ctfrefine_test/
zarr-particle-polish local \
  --particles-starfile refined_particles.star \
  --tomograms-starfile tomograms.star \
  --ref1 half1.mrc --ref2 half2.mrc \
  --box-size 384 --do-motion \
  --output-dir tests/output/sample_polish_test/

The data-portal and copick-data-portal variants write tomograms.star to <output-dir>/input. Supply the refined particles and half-maps:

zarr-particle-ctfrefine data-portal \
  --dataset-ids 10426 \
  --particles-starfile refined_particles.star \
  --ref1 half1.mrc --ref2 half2.mrc \
  --box-size 384 --do-defocus \
  --output-dir tests/output/sample_ctfrefine_portal/

To generate only the tomograms.star (and feed it to the local variants yourself):

zarr-particle-tomograms data-portal --dataset-ids 10426 --output-dir tests/output/sample_tomograms_test/

Import portal runs for a RELION project

zarr-particle-importtomo writes a RELION 5 tomogram set whose tilt series stay on S3: tomograms.star, one tilt-series star per tomogram and sparse placeholder MRCs. It is also the zarrparticletools.importtomo pipeliner job. The set is for zarr-particle-tools programs only: stock RELION programs that read pixels (particle reconstruction, CTF refinement, polishing, subtomogram extraction) read the all-zero placeholder, and e.g. relion_tomo_reconstruct_particle returns an all-zero map without an error. Use zarr-particle-export for a project with the tilt series on disk.

For each run the import resolves one tomogram, which fixes the alignment and voxel spacing:

  • --tomogram-type default|raw|filtered|denoised (default = the run's visualization default), narrowed by --tomogram-software, --reconstruction-method and --voxel-spacing, or pinned with --tomogram-ids. A run with several matching tomograms is refused with the candidates listed.
  • The tables come from the same builder as zarr-particle-tomograms: rlnTomoName is run_<id>_tiltseries_<id>_alignment_<id>_spacing_<id>, one optics group per tilt series, rlnTomoSizeX/Y/Z at voxel spacing = tilt-series pixel size, rlnTomoImportFractionalDose from the portal's exposure dose and rlnTomoTomogramBinning. STAR paths are relative to the working directory (the project).
  • A section that cannot be written (excluded by the alignment, e.g. an AreTomo dark frame, or with an incomplete CTF, alignment or dose record) keeps its source index in N@placeholder and is listed with its reason in portal_selection.json. Runs with an affine transformation are refused. A LOCAL alignment is written as its global per-tilt geometry with a warning, and portal_selection.json records each run's alignment_type.
  • --selection portal_selection.json imports a stored selection, and fails if the portal changed since it was resolved (including per-tilt values) or if it was resolved on the other portal (--staging). zarr_particle_tools.generate.portal_selection.resolve() produces one without the extraction stack.
  • --amplitude-contrast (default 0.07) and --hand (default -1) set what the portal does not record.
zarr-particle-importtomo data-portal --dataset-ids 10426 --run-ids 16848,16849 --output-dir Import/job001

Export an on-disk project

zarr-particle-export creates a portable project for use without portal access or with stock RELION. It downloads the complete tilt-series stacks, which can be large, updates the tilt stars to use local MRC files, and removes tomoTiltSeriesURI.

zarr-particle-export data-portal \
  --dataset-id 10426 \
  --annotation-name ribosome --inexact-match --ground-truth \
  --output-dir 10426_ondisk

Testing

Strict extraction and reconstruction comparisons against RELION 5.0 check every voxel with magnitude-aware float32 tolerances and report the worst mismatch. Broader float16 and experimental cases use looser comparisons appropriate to their precision and noise.

  • test_extract_strict.py, test_reconstruct.py — strict per-voxel equivalence vs RELION on synthetic and real data, across binning, cropping, and no-CTF cases.
  • test_ctfrefine.py, test_polish.py (need RELION binaries) — temporarily staged Zarr data matches stock RELION, with two-phase parity coverage for defocus, scale, aberration, and motion modes.
  • tests/unit/ (no RELION needed) — CTF envelope and phase shift, dose frequency cutoff vs RELION's findDoseXRanges, the zarr readers, and temporary-file preflight/fallback/cleanup safeguards.
uv sync --locked --extra dev
mkdir -p tests/data
(
cd tests/data
for f in zarr_particle_tools_test_data_large zarr_particle_tools_test_data_small; do
  curl -L --fail --retry 5 --retry-delay 5 --continue-at - -o "$f.tar.gz" \
    "https://zenodo.org/records/21797999/files/$f.tar.gz?download=1"
done
for f in *.tar.gz; do tar -xzf "$f"; done
)

The two archives are about 9.34 GB compressed and need additional space when extracted. The record ID is also set as ZENODO_RECORD in .github/workflows/pytest.yml and .github/workflows/pytest_full.yml.

On shared or login nodes, avoid pytest -n auto; it starts one worker per core and can oversubscribe the CPU through BLAS thread pools. Use a fixed worker count:

OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 uv run --locked pytest -n 4 -q

Known limitations

If you would like to see a feature added, on or off this list, please open an issue.

Local and copick jobs refuse, before reading tilt series, what they would not do as RELION does, naming the option or column and file. The pipeliner jobs report the same from validate_joboptions(). For extraction and reconstruction this covers the options listed below, a non-empty other_args and angles left blank in some rows; for every job, tilt series it cannot read. Particle reconstruction writes the maps only, like RELION's; the internal extraction it runs is removed with its star files.

Subtomogram orientations (rlnTomoSubtomogram*, e.g. a filament frame) are applied as RELION applies them: extraction rotates the origin offsets by A_sub and carries every particle column through to particles.star; reconstruction projects with A_sub · A_part.

Extraction and reconstruction

  • Extraction lacks RELION's --apply_orientations mode (writing A_sub · A_part into the subtomogram angles and zeroing rlnAngle*); subtomogram orientations themselves are applied (see above)
  • Does not apply higher-order optical aberrations: extraction refuses non-zero rlnEvenZernike in the optics table when it applies the CTF (the only terms relion_tomo_subtomo uses); reconstruction refuses any non-zero rlnEvenZernike or rlnOddZernike
  • Does not support whitening (power spectral flattening)
  • Does not support --only_do_unfinished

Extraction

  • Does not support 3D volume extraction or write any *.mrcs files other than the 2D stacks themselves
  • Does not support min_frames or max_dose (zarr-particle-pipeline rejects --max-dose before extraction)
  • Does not support grid precorrection
  • Does not support cone flags (--cone_weight, --cone_angle, --cone_sig0)

Reconstruction

  • Does not support helical symmetry
  • Only reproduces RELION's --no_circle_crop mode; its default circle cropping is not implemented
  • Does not support --dont_apply_offsets (extraction supports --dont-apply-offsets)
  • Requires a single rlnTomoTiltSeriesPixelSize across optics groups
  • Does not support weight_*.mrc output files
  • Does not support RELION's reconstruction backup / --no_backup behavior

Development

conda create -n zarr-particle-tools python=3.12
conda activate zarr-particle-tools
pip install uv

git clone git@github.com:czimaginginstitute/zarr-particle-tools.git
cd zarr-particle-tools
uv sync --locked --extra dev

License

zarr-particle-tools is distributed under the terms of the MIT license.

Code of Conduct

This project adheres to the Contributor Covenant code of conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to opensource@chanzuckerberg.com.

Reporting Security Issues

If you believe you have found a security issue, please responsibly disclose by contacting us at security@chanzuckerberg.com.

Metadata

Release files for zarr-particle-tools 1.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for zarr-particle-tools 1.1.0
File Size Uploaded Provenance Actions
zarr_particle_tools-1.1.0.tar.gz 1.5 MB With provenance Details for zarr_particle_tools-1.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for zarr-particle-tools 1.1.0
File Interpreter ABI Platform Actions
zarr_particle_tools-1.1.0-py3-none-any.whl Python 3 none any Details for zarr_particle_tools-1.1.0-py3-none-any.whl

Total release size: 1.6 MB

Release files / zarr_particle_tools-1.1.0.tar.gz

Download URL zarr_particle_tools-1.1.0.tar.gz
Size 1.5 MB
Tags Source
SHA-256 checksum
How to use checksums
b988d43ffb5591bc632de57562c77b44478c08a8df1118d51f346ed9882d6c07
BLAKE2b-256 checksum
How to use checksums
52e204eb6d71020b3643c240bc3fa83523b6b58862a820dfe20e06ba5de1448e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 3, 2026.

Transparency log

Release files / zarr_particle_tools-1.1.0-py3-none-any.whl

Download URL zarr_particle_tools-1.1.0-py3-none-any.whl
Size 143.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
d84139ff63df3f74bad84948151b43cb07259d02a314cf57d72bc6e0ad03e184
BLAKE2b-256 checksum
How to use checksums
bd99257af4d42b309d69b5441e04ab6abaa3ddb3b958974e67bfbe2fc851da95
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 3, 2026.

Transparency log

Security for zarr-particle-tools 1.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for zarr_particle_tools-1.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 3, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page