sam-dealer 0.1.0a1

High-throughput, zero-IO parallel dispatcher for SAM/BAM streams. Distributes reads by index (round-robin) to persistent workers with backpressure management.

sam-dealer

Stream Parallelization for Genomic Alignments

sam-dealer is a command-line utility designed to transparently parallelize single-threaded steps in SAM/BAM/CRAM pipelines. It functions by splitting a stream of alignment data into contiguous record batches and distributing them across a pool of persistent worker processes.

The Problem: Serial Bottlenecks & Engineering Complexity

While many bioinformatics tools that use SAM/BAM/CRAM files as input support multithreading for compression, the downstream processing logic—parsing, filtering, or conversion—often runs on a single core. Parallelizing these tasks is historically difficult for two reasons:

1. Algorithmic Rigidity: Many critical algorithms, such as pairtools parse to convert alignments to Hi-C pairs, strictly require name-sorted inputs. This requirement renders standard parallelization strategies—such as splitting by genomic region (chromosome)—impossible, as the data is not spatially ordered.

2. Orchestration & Testing: Implementing a custom multiprocessing dispatcher is error-prone and difficult to load balance, often leading to "straggler" processes that delay the entire pipeline. Furthermore, tightly coupling complex dispatch logic with scientific analysis code violates separation of concerns, making the core biological algorithms fragile and difficult to test in isolation.

The Solution

sam-dealer resolves these bottlenecks by handling the parallelization infrastructure so that it can be separated from the analysis logic. It multiplexes the alignment stream in memory, reading records once and "dealing" them to worker processes in a round-robin fashion.

This approach:

• Preserves Sort Order: Validates and replicates SAM headers, ensuring subprocesses receive valid streams that respect the input order (name-sorted or otherwise).

• Guarantees Load Balancing: Round-robin distribution ensures even work allocation regardless of coverage density.

• Simplifies Development: Allows bioinformaticians to write simple, single-threaded, testable scripts that read from standard input, while sam-dealer handles the scaling automatically.

Origin & Performance

The development of sam-dealer was motivated by optimization needs within the Hich pipeline, where pairtools parse presented a significant bottleneck. It also addresses architectural complexities observed in tools like ScaleMethyl's methylation calling script, where custom, ad-hoc dispatch methods proved difficult to scale robustly.

In benchmarks targeting pairtools parse2, sam-dealer demonstrated the ability to scale throughput linearly with core count until reaching the physical limits of system memory bandwidth, achieving a 7.8x speedup on a standard 12-core system.

Author Developed by Ben Skubi, Adey and Yardimci Labs, Oregon Health & Science University (OHSU) Contact Use github issues page for sam-dealer. License MIT Citation If you use sam-dealer, please cite the software archive:

• Ben Skubi. (2026). sam-dealer: Stream Parallelization for Bioinformatics (v1.0). Zenodo.

Installation

mamba install samtools parallel mbuffer click
pip install sam-dealer

Examples

Split input.bam into 10 .bam files named 0.bam, 1.bam, etc.

sam-dealer --jobs 10 -o {job}.bam input.bam "samtools view -b"

Spins up 10 persistent pairtools parse jobs. Job 0 writes to 0.pairs, job 1 to 1.pairs, etc. Streams in input.bam, splitting into batches of about 100,000 consecutive records. The first batch is piped into job 0, the second into job 1, continuing in a round-robin until the input is fully processed.

sam-dealer --jobs 10 -o {job}.pairs input.bam "pairtools parse -c hg38.sizes"

CLI Reference

Usage: sam-dealer [OPTIONS] [INPUT_SOURCE] COMMAND

Positional Arguments

• INPUT_SOURCE The path to a BAM, SAM, or CRAM file to read. Use - to read from standard input (stdin). Note: If reading from stdin, the --header option is required.

• COMMAND The command string to run in parallel (e.g., "pairtools parse2 -c genome.sizes"). Note: This command will receive SAM-formatted (plaintext) data on its standard input.

Option	Default	Description
-j, --jobs INT	1	Number of parallel worker processes to spawn. Default: 1.
-o, --output TEMPLATE		Output filename template for each job. Use {job} as a placeholder for the job index (0, 1, ...).Example: -o results/part_{job}.pairs.gz.
-e, --error TEMPLATE		Stderr filename template for capturing job logs.Example: -e logs/job_{job}.err.
-b, --batch-size TEXT	100000	Size of contiguous data chunks sent to workers. Supports integer values for line count or size strings for bytes.Examples: 100000 (lines), 10MB, 1GB. Default: 100000 lines.
-t, --threads INT	4	Number of samtools threads used for input decompression (only applies when INPUT_SOURCE is a file). Default: 4.
--header FILE		Path to a SAM/BAM/CRAM file containing the header. Required only when INPUT_SOURCE is stdin (-), as the header cannot be reliably extracted from a raw stream.
-h, --help		Show the help message and exit.
--write-buffer INT	31457280 (30MB)	Size (in bytes) of the buffer used to write data to the subprocess stdin. Increasing this may reduce system call overhead on extremely high-throughput systems.
--mbuffer-size TEXT	1G	Size of the buffer for the internal mbuffer process (if available). Controls how much data can be buffered in memory before blocking the reader.
--mbuffer-opts TEXT	-t -q	Additional flags passed directly to the mbuffer command. Can be specified multiple times.
--line-size INT	300	Estimated size of a single SAM record in bytes. Used to calculate read buffers when --batch-size is specified in lines (rather than bytes).

Note on Batch Sizes The --batch-size argument determines the granularity of the round-robin distribution.

Line-based (100000): Easier to reason about for record counts. sam-dealer guarantees batches always end on a newline. sam-dealer does not guarantee that every batch gets precisely the number of lines - it is an approximation only.

Byte-based (10MB): sam-dealer will read precisely this many bytes and then read until the next newline to complete the record (up to EOF).

FAQs

How fast is sam-dealer?

Very fast: sam-dealer's throughput is nearly that of a standard Linux pipe. For single-threaded commands, throughput tends to scale nearly linearly with the number of jobs until saturation. Overall throughput will typically be bottlenecked by the command you are parallelizing, not by sam-dealer's rate of data transfer into the job.

Can I use a pipeline as the command?"

Yes! You can use any valid Linux command that can stream from stdin.

What guarantees does sam-dealer make about the order of the input and output?

sam-dealer guarantees that lines batched to jobs are received in the original sort order within the batch, that consecutive batches are distributed to consecutive jobs in the round-robin, and that input and output lines are not truncated.

Can sam-dealer modify the alignment records it passes through?

No. For performance reasons, sam-dealer does not parse the alignment data itself. It is a highly optimized but fundamentally "dumb" dispatcher. However, you can use tools like pysam or samtools in the command as part of a pipeline to modify records within the job.

Can sam-dealer ensure alignment blocks with the same read name are dispatched to the same job?

No, but the 100,000-record default batch size makes data loss due to name block splitting an insignificant practical issue for most applications, since only 0.002% of consecutive records are dispatched to separate processes. If need be, the batch size can be increased (--batch-size).

Can sam-dealer combine the job outputs?

Generally, it's recommended to use the -o option so that each process writes to a separate output file. However, if no output (-o) option is given, sam-dealer streams streams all jobs to stdout, mixing lines together from different jobs. The only guarantee it makes is that every plaintext line of output from a job will be written intact. This might be acceptable if output order is irrelevant, or when following up with custom cleanup code.

Does using sam-dealer risk OOM errors?

Each job is backed by a configurable mbuffer that spills to disk if there is too much memory pressure. However, parallelizing 10 processes does mean they will typically consume 10x the RAM.

Can sam-dealer batch by genomic region?

Not for now.

Why not use samtools split for this?

samtools split splits SAM/BAM/CRAM files by a read group or tag and produces new files on disk. This is write amplification, introduces disk I/O latency, and requires you to implement a pipeline over all the input files and clean up. sam-dealer lets you split by consecutive blocks of lines, streams in RAM, and simplifies parallelizing the command that processes the split.

Why not use GNU parallel for this?

sam-dealer is designed for high-throughput dataflow, with data transfer speeds from stdin to each job approaching that of a standard Linux pipe. GNU parallel's data throughput is much slower. Streaming large volumes of data through GNU parallel tends to result in worse performance than single-threaded tools.