zdata-py 0.1.3

Efficient storage and access for large single-cell RNA datasets (supports Zarr and H5AD formats)

zdata

Efficient sparse matrix storage and retrieval for large-scale transcriptomics datasets using seekable Zstandard compression.

---

What is zdata?

zdata is a Python library designed for bioinformaticians working with large single-cell RNA-seq datasets. It allows you to:

• Store large datasets efficiently using block-based compression
• Query specific cells or genes quickly without loading entire matrices into memory
• Convert existing datasets from Zarr or H5AD (AnnData) into a fast, queryable format

Think of zdata as a compressed, random-access backend for single-cell expression data.

---

Overview

zdata uses a custom on-disk format optimized for sparse single-cell data:

• Block-compressed CSR layout
  Expression data are grouped into small row blocks (e.g. 16 cells per block) for fast access.

• Seekable Zstandard compression
  Enables random access without decompressing entire files.

• Chunked storage
  Large datasets are split into multiple chunk files (e.g. 8192 cells per file), enabling scalability to millions of cells.

This design provides excellent compression ratios while maintaining fast random access.

---

Features

• Fast random access to individual cells
• Efficient disk compression
• Scales to millions of cells and genes
• Simple Python API
• Supports Zarr and H5AD (AnnData) inputs
• Automatic detection of mixed input formats
• Optional column-major (gene-based) queries

---

Installation

Recommended: Install from PyPI

zdata can be installed directly from PyPI and does not require manual compilation of ZSTD.

pip install zdata-py

📦 PyPI package: http://pypi.org/project/zdata-py/

---

Installing from source (developers only)

git clone <repository-url>
cd zdata
pip install -e .

---

Quick Start Guide

Building a zdata Dataset

zdata builds a compressed dataset from a directory containing:

• Zarr directories (*.zarr/)
• H5AD / HDF5 files (*.h5, *.hdf5, *.h5ad)

Mixed formats are supported.

Validate input data

python -m zdata.build_zdata.check_directory /path/to/data

---

Build the dataset

from zdata import build_zdata_from_zarr

zdata_dir = build_zdata_from_zarr(
    zarr_dir="/path/to/data",
    output_name="my_dataset.zdata",
    block_rows=16,
    max_rows=8192,
    obs_join_strategy="outer"
)

---

Opening a Dataset

from zdata import ZData

reader = ZData("my_dataset.zdata")
print(reader.shape)

---

Querying Data

Row-based (cells)

adata = reader[0:100]
adata = reader[[0, 10, 20]]

Raw sparse matrix:

csr = reader.read_rows_csr([0, 100, 200])

---

Column-based (genes)

Requires column-major (X_CM) support.

matrix = reader["GAPDH"]
matrix = reader[["GAPDH", "PCNA"]]

---

Memory Estimation

estimate = reader.estimate_memory_requirements(row_indices=[0, 1, 2])
print(estimate["estimated_memory_gb"])

---

Command-Line Tools

Convert MTX to zdata:

./ctools/mtx_to_zdata matrix.mtx output_name

Read rows:

./ctools/zdata_read --binary output_name.zdata/0.bin "100,200,300"

---

Project Structure

zdata/
├── core/
├── build_zdata/
├── ctools/
├── files/
└── tests/

---

Testing

pytest tests/

---

Performance

Optimized for random-access cell queries, compression efficiency, and scalability to millions of cells.

---

License

See the LICENSE file for details.