scigantic_empiar

Explore EMPIAR — EMBL-EBI's public archive of raw cryo-EM / cryo-ET image data (~3,000 datasets, ~8.9 PiB) — from Python, without downloading anything.

EMPIAR is served over EBI's public HTTPS at ~1.5 MB/s per connection. scigantic_empiar parallelises HTTP range reads (8-way ≈ 5–10 MB/s) so you can pull a single frame from a many-GB entry in seconds, decode the MRC, and render the micrograph + its power spectrum — nothing is copied to disk.

import scigantic_empiar as se

se.preview(10406)                      # render the micrograph below, in seconds
se.EmpiarClient().summary(10406)       # title, pixel size, method, DOI, EMDB/PDB cross-refs
se.EmpiarCatalog().search("ribosome")            # search the whole archive (instant)
se.EmpiarCatalog().search("GPCR", max_res=3.0)   # by science, not just by title

One frame of EMPIAR-10406 (a 70S-ribosome dataset) pulled straight from EBI over parallel range reads — the carbon-foil edge, ice, and particles are visible at left; the FFT is at right. Nothing was downloaded to disk.

Install

pip install "scigantic-empiar[viz]"

Core (numpy, requests) is enough for the readers; [viz] adds matplotlib / pandas / pillow for preview() and the catalog gallery.

What it does

Why parallel range reads

EBI throttles per connection (~1.5 MB/s) and past ~8 concurrent connections. pread splits a read into ~8 concurrent range requests, which aggregates to ~5–10 MB/s — enough to look at any entry interactively. For heavy reprocessing of a whole multi-hundred-GB dataset, mirror it to fast storage first (add_to_fast_workspace); streaming a full entry at 1.5 MB/s isn't practical.

Existing work

The job splits in two: parse MRC, and read bytes from a remote file. Both have existing libraries; neither covers the specific case here.

• mrcfile (CCP-EM) is the standard MRC reader. Its lazy mode is a numpy memmap, which needs a local filesystem path — it does not issue HTTP range requests. scigantic_empiar parses the 1024-byte header directly (parse_mrc_header) to seek to one frame of a remote file without a local copy.
• fsspec HTTPFileSystem turns byte reads into HTTP range requests and can fetch many ranges concurrently (cat_ranges). pread is a small equivalent, kept dependency-free and tuned to EBI's ~8-connection throttle; moving the transport onto fsspec is a reasonable later change.
• copick (CZI, Protein Science 2026) is the closest cryo-EM analog: an fsspec-backed, server-less dataset API with lazy reads. It assumes data stored as OME-Zarr (chunked, multiscale). EMPIAR entries are raw MRC/TIFF, so copick needs a per-entry zarr conversion first — the conversion that MRC's flat layout lets scigantic_empiar skip.

Notes

• MRC/MRCS (movies, micrographs, tomograms, particle stacks) and some TIFF. Files often nest a couple subdir levels down; find_mrc handles that.
• Entry ids are opaque numbers — discover datasets by metadata (EmpiarCatalog.search, or the EMPIAR website), not by listing the tree.
• Inside a Scigantic cryo-EM notebook this is preinstalled and the archive is also FUSE-mounted at $SCIGANTIC_MOUNT_PATH; standalone, it streams straight from EBI.

License

MIT.

Search

EmpiarCatalog.search() matches across the dataset title, the deposited sample name, protein and organism names, method and cross-referenced accessions, with a small cryo-EM synonym vocabulary, and filters on size, resolution, molecular weight, half-map/mask availability and EMDB cross-reference.

cat = se.EmpiarCatalog()
cat.search("GPCR", max_res=3.0, max_chain_kda=100)
cat.search("cryoET", min_gb=100, sort="size")
cat.search("ribosome", half_maps=True, sort="resolution")

max_chain_kda is the largest single polymer, which is the filter you want for "a receptor under 100 kDa": the assembled complex carries the G protein and any nanobodies, so it is almost always heavier than the molecule of interest.

Changed in 0.2.0. Before that release search(query=...) matched against the dataset title only. EMPIAR titles describe the experiment ("Cryo electron microscopy of ..."), and the scientific vocabulary people actually search by lives in the EMDB cross-reference, so search("GPCR") returned nothing at all across the whole archive. It now returns 81 entries. If you pinned 0.1.0 because search seemed to return too little, this is why.

Relationship to the Scigantic notebook image

This package and the copy preinstalled in Scigantic's cryo-EM notebook image are the same source. They were not always: the two drifted into forks differing in 23 functions, and the published half kept shipping readers that had already been fixed on the other side, including an MRC-only thumbnail that used stride decimation and could not read TIFF micrographs at all.

0.3.0 collapses that. scigantic_empiar/__init__.py and _search.py here are byte-identical to the shipping copy (see SYNC.md); the small per-topic modules remain as thin re-export shims so from scigantic_empiar.mrc import read_mrc keeps working.