partake 0.5.1

Shard input to persistent jobs at raw pipe speed.

partake

Shard input to persistent parallel jobs at the rate of a pipe.

Example

seq 0 10000000 | partake -n 10 'cat'

Install

CLI version:

pip install partake

With development dependencies:

pip install partake[dev]

Recipes

Split a SAM/BAM/CRAM file into 2 .bam chunks.

samtools view input.bam | partake -n 2 -s 1 -o {id}.bam "bash -c 'cat <(samtools view -H input.bam) - | samtools view -b'"

Here, we convert the input to SAM (plaintext) and pipe the records to stdin. We use the -s 1 option to ensure that lines are not truncated. In the command, we prepend the header to each worker's input record stream and convert back to bam. Output is 0.bam and 1.bam.