count_split

count_split reproducibly allocates observed count molecules into statistically independent analysis folds. It is a Python implementation inspired by Neufeld et al.'s count-splitting approach for avoiding double use of the same counts in single-cell clustering and differential-expression analysis.

• Paper: Inference after latent variable estimation for single-cell RNA sequencing data
• Reference R package and methodology: countsplit

Please cite the original methodology when using this package.

Allocation contract

For every observed molecule, the implementation draws one uniform random value and assigns that molecule to exactly one requested fold. Multi-fold allocation is simultaneous: [0.325, 0.325, 0.35] means those final probabilities, not a sequence of percentages applied to a shrinking remainder.

The implementation guarantees:

• element-wise count conservation across all output folds;
• deterministic results for the same seed and canonically ordered matrix;
• identical seeded allocation for equivalent dense, CSR, and CSC inputs;
• no sparse-to-dense conversion;
• preservation of integer input dtypes on output (CSC for every sparse input format); and
• explicit rejection of negative, non-finite, fractional, boolean, complex, or out-of-range counts.

Installation

From a checkout:

python -m pip install .

For development:

python -m pip install -e .
python -m pytest

Python 3.10 or newer is required.

Usage

The legacy API expects variables in rows and samples in columns. For a single-cell matrix, that means genes × cells.

import numpy as np
from count_split.count_split import multi_split

counts = np.random.default_rng(7).negative_binomial(
    2,
    0.4,
    size=(1_000, 500),
)

train, validation, test = multi_split(
    counts,
    percent_vect=[0.325, 0.325, 0.35],
    seed=123456,
)

assert np.array_equal(train + validation + test, counts)

Weights are normalized, so [1, 1, 2] is equivalent to [0.25, 0.25, 0.5]. A zero-weight fold is retained as an all-zero result. multi_split continues to return a list for backward compatibility.

Sparse matrices

CSR, CSC, and other SciPy sparse inputs are accepted. Duplicate sparse coordinates are summed once in safe int64 scratch space. All sparse outputs are canonical CSC matrices and preserve the input integer dtype when it can represent the canonicalized logical counts.

from scipy import sparse
from count_split.count_split import multi_split

counts_csr = sparse.csr_matrix(counts)
train, validation, test = multi_split(
    counts_csr,
    percent_vect=[0.325, 0.325, 0.35],
    seed=123456,
)

assert ((train + validation + test) != counts_csr).nnz == 0

AnnData

AnnData normally stores cells in rows and genes in columns, so transpose X for this legacy API:

train, validation, test = multi_split(
    adata.X.T,
    percent_vect=[0.325, 0.325, 0.35],
    seed=123456,
)

The returned matrices are genes × cells. Downstream code that expects the AnnData orientation should transpose each result back.

Two-fold APIs

The original pairwise functions and argument order remain available. The new seed parameter is appended, so existing positional calls remain valid.

from count_split.count_split import split_mat_counts, split_sparse

dense_a, dense_b = split_mat_counts(counts, percent_1=0.4, seed=17)
sparse_a, sparse_b = split_sparse(counts_csr, percent_1=0.4, seed=17)

Calling seed_rng(value) before a call that omits seed remains supported for legacy integrations. New code should use the explicit seed argument. Valid explicit seeds are integers from 0 through 4,294,967,295.

Dense HDF5

split_mat_counts_h5 preserves the original two-output file interface and copies all non-target HDF5 content into both outputs. Only the selected dense, two-dimensional dataset is replaced with split counts.

from count_split.count_split import split_mat_counts_h5

split_mat_counts_h5(
    "input.h5",
    "fold_a.h5",
    "fold_b.h5",
    percent_1=0.5,
    bin_size=5_000,
    key="infile",
    seed=17,
)

Memory and reproducibility notes

All cumulative offsets and total-molecule bookkeeping use int64 internally, but this does not force int64 storage on output. Each output entry is bounded by its corresponding canonicalized input count. Integral real-valued inputs are accepted for legacy compatibility but return integer (int64) folds.

The chosen non-parametric algorithm explicitly materializes one uniform value and one fold index per observed molecule, plus a folds × nonzero-entry re-collation buffer. Its peak memory therefore scales with the total molecule count, not only matrix dimensions or sparse nonzeros. This is deliberate and should be capacity-planned for very deep matrices.

For ordinary dense and sparse calls, bin_size is accepted only for signature compatibility and does not change matrix-wide allocation. The HDF5 function uses bin_size to process contiguous column chunks while maintaining one RNG stream; seeded output is invariant to that chunk size.

Reproducibility requires the same package/runtime stack, input values, matrix shape, row/column order, fold weights, and seed. Equivalent dense, CSR, and CSC representations are canonicalized to the same column-major allocation order.

Compatibility

Version 1.0.0 retains the original public function names, existing positional arguments, HDF5 behavior, CSC sparse return convention, and list return from multi_split. Correctness-related behavior changes are documented in CHANGELOG.md.

License and contact

Licensed under the GNU Affero General Public License v3.0; see LICENSE.

Source: github.com/scottyler89/count_split

Repository owner: scottyler89@gmail.com