picklerick

Python bindings for SCX, a Rust engine that converts single-cell data between formats in bounded memory.

pip install scx-picklerick

import picklerick as pk

The PyPI distribution is named scx-picklerick because picklerick is too close to an existing project name. The import name is picklerick.

Requires Python 3.14 or newer. Wheels are built for Linux (x86_64, aarch64) and macOS (arm64) and include a statically linked HDF5. You don't need a system HDF5.

Early release. The API below is the one meant to stay, but it may still change. Versions follow the scx engine. Please report problems at https://github.com/btraven00/scx/issues.

What it does

Supported inputs: H5AD, H5Seurat (dgCMatrix or BPCells-backed), BPCells directories, 10x HDF5, MatrixMarket, and AnnData Zarr. Engine errors are raised as pk.PickleRickError.

convert streams the matrix in chunks of chunk_size cells, so peak memory does not grow with the number of cells. When converting to H5Seurat from a source that has a counts layer, X is treated as normalised data: X is written to the Seurat data slot and the counts layer to the counts slot.

Examples

import picklerick as pk

pk.convert("pbmc3k.h5seurat", "pbmc3k.h5ad")
pk.convert("pbmc3k.h5ad", "pbmc3k.h5seurat")

adata = pk.read("pbmc3k.h5seurat")          # AnnData
pk.write_h5seurat(adata, "copy.h5seurat")

pk.inspect("atlas.h5ad")["n_obs"]

Streaming

open_stream yields CSR row blocks without loading the whole matrix or building an AnnData object. Decoding runs on a background thread a few chunks ahead of the consumer. The numpy arrays take ownership of the decoded buffers, so there is no copy per chunk.

import numpy as np
import picklerick as pk

gene_sums = None
for chunk in pk.open_stream("atlas.h5ad", chunk_size=5000):
    if gene_sums is None:
        gene_sums = np.zeros(chunk.n_vars)
    gene_sums += np.bincount(chunk.indices, weights=chunk.data, minlength=chunk.n_vars)

For deflate-compressed H5AD, chunks are decompressed on all cores. HDF5's own filter pipeline would decompress them on a single core.

Peak RSS and wall time for a per-gene-sum workload (bench/python/, release build, chunk 5000, 16 cores):

Peak memory grows with chunk_size: HLCA uses 0.55 GB at 5000 and 3.6 GB at 50000.

Development

From the repository root:

pixi run -e py test-picklerick            # debug build + pytest
pixi run -e py install-picklerick-release # before any benchmarking

A debug build is about 3× slower at HDF5 decoding, so use the release build for timings. Tests use the small fixtures tracked in git. Tests that need the large local fixtures (pixi run -e test fixtures) are skipped, with the reason shown, when those fixtures are missing.

License

MIT.