zarr-particle-tools
zarr-particle-tools provides a full subtomogram averaging (STA) pipeline and standalone jobs for
subtomogram extraction, reconstruction, CTF refinement, and Bayesian polishing. Tilt series can be read
from local or S3-backed MRC files and OME-Zarr,
including datasets in the CryoET Data Portal.
Extraction and reconstruction are Python implementations of the corresponding
RELION 5 jobs and stream Zarr pixels without first downloading the
complete tilt series. CTF refinement and polishing run RELION against Zarr through temporary MRC
files, preferring /dev/shm and falling back to the system temp directory. zarr-particle-pipeline
connects the jobs through py2rely for SLURM-based STA
workflows.
Contents
- Quickstart (container)
- Installation
- Pipeline prerequisites
- Commands
- Usage
- Testing
- Known limitations
- Development
Quickstart (container)
relion-zarr-sta-client
installs py2rely and zarr-particle-tools natively (needed for sbatch, which requires the host's
own SLURM setup) and wires them to run RELION and the job commands below through the
relion-zarr-sta container instead of a
local RELION build. On a SLURM cluster, this is enough to run the full pipeline:
apptainer pull relion-zarr-sta.sif oras://ghcr.io/czimaginginstitute/relion-zarr-sta-sif:5.0-cuda12.8
pip install git+https://github.com/czimaginginstitute/relion-docker.git#subdirectory=client
relion-zarr-sta-client --sif relion-zarr-sta.sif --out ~/relion-shims/bin --wire-py2rely
export PATH="$HOME/relion-shims/bin:$PATH"
--wire-py2rely points py2rely's SLURM job scripts at the container automatically; the PATH
export lets you also run any of the commands below directly. See relion-docker's
client/ for details.
Otherwise, see Installation below for a standalone/manual setup.
Installation
conda create -n zarr-particle-tools python=3.12
conda activate zarr-particle-tools
pip install uv
uv pip install zarr-particle-tools
- Extraction and reconstruction need no RELION installation.
- CTF refinement and polishing require the corresponding
relion_tomo_*binaries. - The full pipeline has the additional prerequisites below.
The [pipeliner] extra installs CCPEM pipeliner
for the four registered job wrappers:
uv pip install "zarr-particle-tools[pipeliner]"
Installing py2rely in the same environment also installs ccpem-pipeliner.
Pipeline prerequisites
The Quickstart container already has all of the below; skip this section if you're using it.
zarr-particle-pipeline requires:
- py2rely on
PATH. - RELION 5 binaries on
PATH. Source RELION'ssetup-env.shor add itsbuild/bindirectory. - This package installed in the same environment as py2rely so its
ccpem_pipeliner.jobsentry points are registered. - A SLURM cluster with
sbatchavailable.
Run zarr-particle-pipeline preflight to check py2rely, RELION, ccpem-pipeliner, and job registration.
SLURM availability is not checked. The same preflight runs automatically before pipeline preparation.
Commands
Every command takes --help. Sources are named consistently: local (your own star files), copick-local,
data-portal, copick-data-portal (copick picks + portal tilt series; copick run names must be portal run
IDs).
| Command | Purpose | Sources |
|---|---|---|
zarr-particle-pipeline |
Full STA pipeline via py2rely | local, copick-local, data-portal, copick-data-portal, plus preflight |
zarr-particle-extract |
Subtomogram extraction (2D stacks) | local, copick-local, data-portal, copick-data-portal |
zarr-particle-reconstruct |
Particle map reconstruction | local, copick-local, data-portal, copick-data-portal |
zarr-particle-ctfrefine |
RELION relion_tomo_refine_ctf on zarr |
local, data-portal, copick-data-portal |
zarr-particle-polish |
RELION relion_tomo_align on zarr |
local, data-portal, copick-data-portal |
zarr-particle-tomograms |
Write a tomograms.star |
data-portal, copick-data-portal |
zarr-particle-importtomo |
Import portal runs into a RELION project (pipeliner: zarrparticletools.importtomo) |
data-portal (IDs or a stored selection) |
zarr-particle-export |
Self-contained on-disk project (downloads tilt series) | data-portal, copick-data-portal |
core/ can also be used directly for projection matrices and point projection, CTF premultiplication,
dose weighting, Fourier cropping, masking, backprojection, interpolation, symmetry, and Zarr/S3 I/O.
Usage
Full pipeline
zarr-particle-pipeline:
- Resolves the selected tilt series and particles.
- Derives the pixel size and cross-checks portal metadata against each tilt-series MRC header.
- Writes star files to
<output-dir>/input. - Runs
py2rely prepare relion5-parametersandpy2rely prepare relion5-pipeline, then submitspipeline.shunless--prepare-onlyis set.
The generated tomograms.star carries a tomoTiltSeriesURI column. py2rely uses that column to select
the Zarr extraction, reconstruction, CTF-refinement, and polishing jobs. Refine3D, Class3D, MaskCreate,
and PostProcess continue to use stock RELION jobs.
The following portal example uses dataset 10426 and the cytosolic ribosome picks from deposition
10358: 5,246 oriented picks across 38 tilt series at 2.165 Å/px, with per-tilt alignment and CTF
metadata.
export TEMPLATE=tests/templates/ribo80s_emd_3883_866_64_resized.mrc
export OUT=10426_ribosome_sta
zarr-particle-pipeline data-portal \
--dataset-ids 10426 \
--deposition-ids 10358 \
--annotation-names "cytosolic ribosome" \
--output-dir "$OUT" \
--reference-template "$TEMPLATE" \
--protein-diameter 330 \
--box-scaling 1.65 \
--num-gpus 4 \
--nthreads 4 \
--prepare-only
--protein-diameteris required.--reference-templateis required unless--run-denovo-generationis set.--run-idslimits the selection to specific runs.- The portal form also accepts deposition, dataset, organism, run, tilt-series, alignment, tomogram, and annotation filters by ID or name.
--pixel-sizeoverrides the derived pixel size;--pixel-size-toladjusts the MRC-header check.--timeoutis the per-job limit in hours;--num-dayssets the SLURM walltime request.--run-class3d --nclasses N --class-selection autoenables 3D classification.--gpu-constraintaccepts one SLURM GPU feature or a comma- or pipe-separated set interpreted as OR. py2rely drops requested features that are unavailable on the cluster.--prepare-onlywritesall_sta_parameters.jsonandpipeline.shwithout submitting. Submit later withcd "$OUT" && sbatch pipeline.sh.
Run zarr-particle-pipeline data-portal --help for the complete option list and current defaults.
Star files are written under $OUT/input. RELION job directories such as Extract/, Refine3D/,
Class3D/, CtfRefine/, Polish/, and PostProcess/ are created alongside them as the pipeline runs.
End-to-end runs can take days on SLURM.
The repository includes tests/templates/ribo80s_emd_3883_866_64_resized.mrc, an 80S ribosome map
(EMD-3883) resampled to 8.66 Å/px in a box of 64
(330 × 1.65 / 8.66 = 62.9, rounded up). The path above assumes a source checkout; otherwise set
TEMPLATE to your own reference. Match its voxel size to the dataset's coarsest requested binning.
When the picks live in copick instead, copick-data-portal takes them from there and the tilt series
from the portal (the copick run names must be portal run IDs):
zarr-particle-pipeline copick-data-portal \
--copick-config config.json \
--copick-name ribosome --copick-user-id picker --copick-session-id 1 \
--copick-dataset-ids 10426 \
--output-dir ribosome_sta \
--reference-template template.mrc \
--protein-diameter 330 \
--box-scaling 1.65 \
--num-gpus 4 --nthreads 4
The science options are the same as for data-portal.
For existing star files, both files must be inside --output-dir because py2rely resolves their paths
relative to the project directory. The pixel size is read from rlnTomoTiltSeriesPixelSize. Zarr jobs
are selected only when tomograms.star contains tomoTiltSeriesURI; otherwise py2rely uses stock
RELION jobs.
zarr-particle-pipeline local \
--output-dir my_sta \
--particles-starfile my_sta/input/particles.star \
--tomograms-starfile my_sta/input/tomograms.star \
--protein-diameter 330 \
--reference-template ref.mrc
copick-local builds particles.star from copick picks and uses the supplied tomograms.star for
optics. It writes the particles file to <output-dir>/input:
zarr-particle-pipeline copick-local \
--output-dir my_sta \
--tomograms-starfile my_sta/input/tomograms.star \
--copick-config pick.json \
--copick-name ribosome --copick-user-id user0 --copick-session-id 1 \
--protein-diameter 330 \
--reference-template ref.mrc
Extraction
For RELION projects, --tiltseries-relative-dir can be omitted if you run from the project root.
zarr-particle-extract local \
--particles-starfile tests/data/relion_project_synthetic/particles.star \
--tomograms-starfile tests/data/relion_project_synthetic/tomograms.star \
--tiltseries-relative-dir tests/data/relion_project_synthetic/ \
--output-dir tests/output/sample_local_test/ \
--box-size 16 --bin 4
Extract a larger box and crop it back down with --crop-size (which must be even and no larger than
--box-size), --no-ctf to skip CTF premultiplication, and --no-circle-crop for noisier real data:
zarr-particle-extract local \
--particles-starfile tests/data/relion_project_unroofing/particles.star \
--tomograms-starfile tests/data/relion_project_unroofing/tomograms.star \
--tiltseries-relative-dir tests/data/relion_project_unroofing/ \
--output-dir tests/output/sample_local_test/ \
--box-size 64 --bin 1 --no-ctf --no-circle-crop
Polished RELION-5 projects can carry per-tilt 2D image deformations (tomo Bayesian polishing with
"model 2D deformations": rlnTomoDeformationGridSizeX/Y and rlnTomoDeformationType in tomograms.star,
rlnTomoDeformationCoefficients in each tilt-series star). They are applied to every projected particle
position exactly as relion_tomo_subtomo does (spline, linear and Fourier models), on top of any
--trajectories-starfile motion; pass --ignore-deformations to extract without them. Skipping them on a
project that was refined with them mis-centres the crops by the local field value, typically tens of Å.
zarr-particle-reconstruct local / copick-local extract through the same step, so they apply deformations
too, matching relion_tomo_reconstruct_particle, and take the same flag.
From the portal without first materializing the tilt series locally:
zarr-particle-extract data-portal \
--run-id "16848, 16851" \
--annotation-names "ribosome" \
--ground-truth --inexact-match \
--output-dir tests/output/sample_data_portal_test/ \
--box-size 128 --bin 2
Reconstruction
zarr-particle-reconstruct local \
--particles-starfile tests/data/relion_project_unroofing/reconstruct_particles.star \
--tomograms-starfile tests/data/relion_project_unroofing/tomograms.star \
--tiltseries-relative-dir tests/data/relion_project_unroofing/ \
--output-dir tests/output/sample_local_reconstruct_test/ \
--box-size 384 --crop-size 256
Reconstruction applies 2D deformations as relion_tomo_reconstruct_particle does.
CTF refinement and polishing
- CTF refinement runs
relion_tomo_refine_ctf; polishing runsrelion_tomo_align. - Zarr tilt series are streamed into temporary MRC files for RELION to read, in a job-owned directory under
--shm-dir(default/dev/shm) that is removed when the job ends, fails or receives SIGTERM (--keep-shmkeeps it). When--shm-diris unusable the system temp directory is used, unless--require-ram-stagingis given: then the job fails before streaming unless--shm-diris tmpfs with room for the staged stacks. This also covers the temporary copies of S3-hosted MRC stacks. The pipeliner jobs require RAM staging by default (do_require_ram_staging). --relion-argspasses additional arguments to every RELION call (the pipeliner jobs'other_args).- Both jobs require a refined
particles.starand reference half-maps from a prior Refine3D job. - There is no
copick-localvariant because raw copick picks are not refined particles. --per-tomogramis the default and uses a staging-bounded two-phase workflow.--all-at-oncestages every selected tilt series together.--n-workers 0selects the worker count automatically.
zarr-particle-ctfrefine local \
--particles-starfile refined_particles.star \
--tomograms-starfile tomograms.star \
--ref1 half1.mrc --ref2 half2.mrc \
--box-size 384 --do-defocus --do-scale \
--output-dir tests/output/sample_ctfrefine_test/
zarr-particle-polish local \
--particles-starfile refined_particles.star \
--tomograms-starfile tomograms.star \
--ref1 half1.mrc --ref2 half2.mrc \
--box-size 384 --do-motion \
--output-dir tests/output/sample_polish_test/
The data-portal and copick-data-portal variants write tomograms.star to <output-dir>/input. Supply
the refined particles and half-maps:
zarr-particle-ctfrefine data-portal \
--dataset-ids 10426 \
--particles-starfile refined_particles.star \
--ref1 half1.mrc --ref2 half2.mrc \
--box-size 384 --do-defocus \
--output-dir tests/output/sample_ctfrefine_portal/
To generate only the tomograms.star (and feed it to the local variants yourself):
zarr-particle-tomograms data-portal --dataset-ids 10426 --output-dir tests/output/sample_tomograms_test/
Import portal runs for a RELION project
zarr-particle-importtomo writes a RELION 5 tomogram set whose tilt series stay on S3: tomograms.star, one
tilt-series star per tomogram and sparse placeholder MRCs. It is also the zarrparticletools.importtomo pipeliner
job. The set is for zarr-particle-tools programs only: stock RELION programs that read pixels (particle
reconstruction, CTF refinement, polishing, subtomogram extraction) read the all-zero placeholder, and e.g.
relion_tomo_reconstruct_particle returns an all-zero map without an error. Use zarr-particle-export for a
project with the tilt series on disk.
For each run the import resolves one tomogram, which fixes the alignment and voxel spacing:
--tomogram-type default|raw|filtered|denoised(default= the run's visualization default), narrowed by--tomogram-software,--reconstruction-methodand--voxel-spacing, or pinned with--tomogram-ids. A run with several matching tomograms is refused with the candidates listed.- The tables come from the same builder as
zarr-particle-tomograms:rlnTomoNameisrun_<id>_tiltseries_<id>_alignment_<id>_spacing_<id>, one optics group per tilt series,rlnTomoSizeX/Y/Zat voxel spacing = tilt-series pixel size,rlnTomoImportFractionalDosefrom the portal's exposure dose andrlnTomoTomogramBinning. STAR paths are relative to the working directory (the project). - A section that cannot be written (excluded by the alignment, e.g. an AreTomo dark frame, or with an incomplete
CTF, alignment or dose record) keeps its source index in
N@placeholderand is listed with its reason inportal_selection.json. Runs with an affine transformation are refused. A LOCAL alignment is written as its global per-tilt geometry with a warning, andportal_selection.jsonrecords each run'salignment_type. --selection portal_selection.jsonimports a stored selection, and fails if the portal changed since it was resolved (including per-tilt values) or if it was resolved on the other portal (--staging).zarr_particle_tools.generate.portal_selection.resolve()produces one without the extraction stack.--amplitude-contrast(default 0.07) and--hand(default -1) set what the portal does not record.
zarr-particle-importtomo data-portal --dataset-ids 10426 --run-ids 16848,16849 --output-dir Import/job001
Export an on-disk project
zarr-particle-export creates a portable project for use without portal access or with stock RELION. It
downloads the complete tilt-series stacks, which can be large, updates the tilt stars to use local MRC
files, and removes tomoTiltSeriesURI.
zarr-particle-export data-portal \
--dataset-id 10426 \
--annotation-name ribosome --inexact-match --ground-truth \
--output-dir 10426_ondisk
Testing
Strict extraction and reconstruction comparisons against RELION 5.0 check every voxel with magnitude-aware float32 tolerances and report the worst mismatch. Broader float16 and experimental cases use looser comparisons appropriate to their precision and noise.
test_extract_strict.py,test_reconstruct.py— strict per-voxel equivalence vs RELION on synthetic and real data, across binning, cropping, and no-CTF cases.test_ctfrefine.py,test_polish.py(need RELION binaries) — temporarily staged Zarr data matches stock RELION, with two-phase parity coverage for defocus, scale, aberration, and motion modes.tests/unit/(no RELION needed) — CTF envelope and phase shift, dose frequency cutoff vs RELION'sfindDoseXRanges, the zarr readers, and temporary-file preflight/fallback/cleanup safeguards.
uv sync --locked --extra dev
mkdir -p tests/data
(
cd tests/data
for f in zarr_particle_tools_test_data_large zarr_particle_tools_test_data_small; do
curl -L --fail --retry 5 --retry-delay 5 --continue-at - -o "$f.tar.gz" \
"https://zenodo.org/records/21797999/files/$f.tar.gz?download=1"
done
for f in *.tar.gz; do tar -xzf "$f"; done
)
The two archives are about 9.34 GB compressed and need additional space when extracted. The record ID
is also set as ZENODO_RECORD in .github/workflows/pytest.yml and .github/workflows/pytest_full.yml.
On shared or login nodes, avoid pytest -n auto; it starts one worker per core and can oversubscribe the
CPU through BLAS thread pools. Use a fixed worker count:
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 uv run --locked pytest -n 4 -q
Known limitations
If you would like to see a feature added, on or off this list, please open an issue.
Local and copick jobs refuse, before reading tilt series, what they would not do as RELION does, naming the option
or column and file. The pipeliner jobs report the same from validate_joboptions(). For extraction and
reconstruction this covers the options listed below, a non-empty other_args and angles left blank in some rows;
for every job, tilt series it cannot read. Particle reconstruction writes the maps only, like RELION's; the
internal extraction it runs is removed with its star files.
Subtomogram orientations (rlnTomoSubtomogram*, e.g. a filament frame) are applied as RELION applies them:
extraction rotates the origin offsets by A_sub and carries every particle column through to particles.star;
reconstruction projects with A_sub · A_part.
Extraction and reconstruction
- Extraction lacks RELION's
--apply_orientationsmode (writingA_sub · A_partinto the subtomogram angles and zeroingrlnAngle*); subtomogram orientations themselves are applied (see above) - Does not apply higher-order optical aberrations: extraction refuses non-zero
rlnEvenZernikein the optics table when it applies the CTF (the only termsrelion_tomo_subtomouses); reconstruction refuses any non-zerorlnEvenZernikeorrlnOddZernike - Does not support whitening (power spectral flattening)
- Does not support
--only_do_unfinished
Extraction
- Does not support 3D volume extraction or write any
*.mrcsfiles other than the 2D stacks themselves - Does not support
min_framesormax_dose(zarr-particle-pipelinerejects--max-dosebefore extraction) - Does not support grid precorrection
- Does not support cone flags (
--cone_weight,--cone_angle,--cone_sig0)
Reconstruction
- Does not support helical symmetry
- Only reproduces RELION's
--no_circle_cropmode; its default circle cropping is not implemented - Does not support
--dont_apply_offsets(extraction supports--dont-apply-offsets) - Requires a single
rlnTomoTiltSeriesPixelSizeacross optics groups - Does not support
weight_*.mrcoutput files - Does not support RELION's reconstruction backup /
--no_backupbehavior
Development
conda create -n zarr-particle-tools python=3.12
conda activate zarr-particle-tools
pip install uv
git clone git@github.com:czimaginginstitute/zarr-particle-tools.git
cd zarr-particle-tools
uv sync --locked --extra dev
License
zarr-particle-tools is distributed under the terms of the MIT license.
Code of Conduct
This project adheres to the Contributor Covenant code of conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to opensource@chanzuckerberg.com.
Reporting Security Issues
If you believe you have found a security issue, please responsibly disclose by contacting us at security@chanzuckerberg.com.
Metadata
Release files for zarr-particle-tools 1.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| zarr_particle_tools-1.1.0.tar.gz | 1.5 MB | With provenance | Details for zarr_particle_tools-1.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| zarr_particle_tools-1.1.0-py3-none-any.whl | Python 3 | none | any | Details for zarr_particle_tools-1.1.0-py3-none-any.whl |
Total release size: 1.6 MB
Release files / zarr_particle_tools-1.1.0.tar.gz
| Download URL | zarr_particle_tools-1.1.0.tar.gz |
|---|---|
| Size | 1.5 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b988d43ffb5591bc632de57562c77b44478c08a8df1118d51f346ed9882d6c07
|
|
BLAKE2b-256 checksum How to use checksums |
52e204eb6d71020b3643c240bc3fa83523b6b58862a820dfe20e06ba5de1448e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 3, 2026.
Transparency logRelease files / zarr_particle_tools-1.1.0-py3-none-any.whl
| Download URL | zarr_particle_tools-1.1.0-py3-none-any.whl |
|---|---|
| Size | 143.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d84139ff63df3f74bad84948151b43cb07259d02a314cf57d72bc6e0ad03e184
|
|
BLAKE2b-256 checksum How to use checksums |
bd99257af4d42b309d69b5441e04ab6abaa3ddb3b958974e67bfbe2fc851da95
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 3, 2026.
Transparency logSecurity for zarr-particle-tools 1.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for zarr_particle_tools-1.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 3, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.