fastQpick 0.3.0

Fast and memory-efficient sampling of DNA-Seq or RNA-seq fastq data with or without replacement.

fastQpick

Fast and memory-efficient sampling of DNA-seq or RNA-seq FASTQ data with replacement. Useful for generating bootstrap replicates to estimate technical variance in downstream analyses, and for subsampling large datasets for testing and benchmarking.

---

Installation

Install via PyPI

pip install fastQpick

Install from Source Code

Using pip:

pip install git+https://github.com/pachterlab/fastQpick.git

---

Usage

Command-line Interface

Run fastQpick with a specified fraction and options:

fastQpick [OPTIONS] FASTQ_FILE1 FASTQ_FILE2 ...

Python API

Use fastQpick in your Python code:

from fastQpick import fastQpick

fastQpick(
    input_files=['FASTQ_FILE1', 'FASTQ_FILE2', ...],
    ...
)

---

Documentation

• Command-line Help: Use the following command to see all available options:

  fastQpick --help
  

• Python API Help: Use the help function to explore the API:

  help(fastQpick)
  

---

Tutorials

Two Jupyter notebooks in notebooks/ walk through fastQpick end-to-end:

• intro.ipynb — Getting started on synthetic data. Simulates a small RNA-seq experiment with known transcript abundances, draws bootstrap replicates with replacement (fraction=1.0, replacement=True), and shows that the bootstrap standard errors recover the analytic multinomial sampling error.
• yeast_example.ipynb — Real-data application reproducing Figure 1 of the paper. Bootstraps a paired-end yeast RNA-seq dataset (SRA SRR453566), re-quantifies each replicate with kallisto, and characterizes the bootstrap distribution of the transcript abundance estimates.

---

Features

• Efficient sampling of large FASTQ files.
• Memory efficient - the occurrence vector is sized to the largest per-read count actually drawn (one byte per read in the common case), and low-memory mode further avoids materializing the array of sampled indices.
• Time efficient - streams through the fastq and writes output in batches - generates a full-size (fraction=1, with replacement) bootstrap replicate of a 500M-read FASTQ in ~26 minutes in standard mode, ~56 minutes in low-memory mode, and ~35 minutes in one-pass mode (see Benchmark below).
• Gzip-compressed output by default, using the ISA-L-accelerated isal library to keep compression from bottlenecking the write pass. Pass --disable-gzip (CLI) or disable_gzip=True (Python API) to write plain FASTQ instead.

---

License

fastQpick is licensed under the 2-clause BSD license. See the LICENSE file for details.

---

Contributing

We welcome contributions! Please see the CONTRIBUTING.md file for guidelines on how to get involved.