hapiq — Python library

Python bindings for hapiq, a CLI tool for downloading scientific datasets from GEO, SRA, Zenodo, Figshare, Ensembl, scPerturb, BioStudies, HCA, and VCP with full provenance tracking.

The library implements the HDL (Has Data Locally) pattern: a Dataset object holds your download parameters but only fetches data when you explicitly call fetch(). Repeated calls with skip_existing=True (the default) are instant if the data is already on disk.

Installation

Wheels for Linux (x86_64, aarch64) and macOS 12+ (x86_64, arm64) ship a static hapiq binary of the same version, so no Go toolchain is required.

pip install hapiq

For development, install from the repo:

cd python/
pip install -e .

Usage

Download a dataset

from hapiq import Dataset

ds = Dataset("geo", "GSE133344", include_ext=[".h5ad", ".csv.gz"])

# Inspect metadata without downloading
print(ds.metadata.title)
print(f"{ds.metadata.file_count} files, {ds.metadata.total_size} bytes")

# Download (skips files already present by default)
paths = ds.fetch("/data/geo/GSE133344")
# → [PosixPath('/data/geo/GSE133344/sample.h5ad'), ...]

Search for datasets

from hapiq import search

results = search("geo", "single cell RNA-seq lung cancer", organism="Homo sapiens")
for r in results:
    print(r.accession, r.title)

Full download result with provenance

result = ds.fetch_result("/data/geo/GSE133344")
print(result.witness_file)      # path to hapiq.json provenance file
print(result.bytes_downloaded)  # bytes transferred this run
for f in result.files:
    print(f.path, f.checksum)   # SHA-256 per file

Supported sources

API reference

Dataset(source, id, **filters)

ds.metadata → Metadata

Cached property. Fetches dataset info (title, file count, organisms, etc.) without downloading any files.

ds.fetch(out_dir, *, resume=False, skip_existing=True, dry_run=False) → list[Path]

Download files, return their local paths. With skip_existing=True (default), files already present are not re-downloaded.

ds.fetch_result(out_dir, *, resume=False, skip_existing=True) → DownloadResult

Like fetch() but returns the full result including checksum info, download stats, and the path to the hapiq.json witness file.

search(source, query, *, organism=None, entry_type=None, limit=0, timeout=60) → list[SearchResult]

Environment variables

Development

cd python/
pip install -e .                             # builds hapiq/bin/hapiq; needs Go
pytest tests/ -m "not integration"          # unit tests (no network)
pytest tests/ -m integration                # integration tests (hits GEO)
HAPIQ_DOWNLOAD_TEST=1 pytest tests/ -m integration  # includes actual file download

The bundled binary must report the package version (hapiq/_version.py). After changing Go code, run pip install -e . again to rebuild it, or delete hapiq/bin/hapiq to use hapiq from PATH.

TODO

• Async fetch() variant for use in notebook environments