demultiplexer3

A Python CLI tool for demultiplexing paired-end Illumina sequencing reads using inline tags. Successor to demultiplexer2.

Key improvements over demultiplexer2

• Output-oriented parallelism — workers write individual samples as soon as parsing is done, instead of blocking on the largest input file.
• Cross-product tag validation — forward and reverse tags are checked jointly (concatenated Levenshtein distance), eliminating the infinite-loop risk of separate checks.
• Configurable minimum distance (--min_dist) and optional tag trimming (--trim).
• DAG-based scheduler using concurrent.futures.ProcessPoolExecutor with dynamic job submission.
• Parquet intermediate format with sample identity classified once at parse time, for fast columnar filtering downstream.

Installation

pip install demultiplexer3

Dependencies

• biopython
• pandas
• openpyxl
• pyarrow
• duckdb
• psutil
• tqdm
• isal (faster gzip via Intel ISA-L; falls back to stdlib gzip if unavailable)

Usage

1. Create a primerset

demultiplexer3 create_primerset --name <name> --n_primers <n>

Creates an Excel file with three sheets (general_information, forward_tags, reverse_tags) for the user to fill in primer sequences and tag names/sequences. IUPAC ambiguity codes are supported. Saved as <name>_primerset.xlsx in the current working directory (same as the tagging scheme below).

2. Create a tagging scheme

demultiplexer3 create_tagging_scheme --name <name> --data_dir <dir> --primerset_path <path>

Scans data_dir for .fastq.gz file pairs, loads the primerset, and interactively asks which tag combinations were used. Produces a tagging scheme Excel file where sample names are filled in by the user.

3. Demultiplex

demultiplexer3 demultiplex --primerset_path <path> --tagging_scheme_path <path> --output_dir <dir> [--workers <n>] [--min_dist <d>] [--trim]

How it works

1. Tag validation & extension — Tags are equalized to the same length, IUPAC codes are expanded, and the cross-product of all (fwd+rev) combinations is checked for minimum Levenshtein distance. If the distance is too low, tags are extended base-by-base from the primer sequence.

2. Parse phase — Each input FASTQ pair is read with BioPython's FastqGeneralIterator. The first N bases of each read are extracted as the tag and classified against that input's known tag combinations right away, storing the result as an integer sample index. All data is written to an lz4-compressed Parquet file (a purely transient intermediate — compression here favors speed over ratio).

3. Write phase — For each sample, DuckDB filters the Parquet file on that integer index and writes matching reads in batches to gzipped FASTQ output files.

4. Cleanup — Parquet files are deleted once all downstream write jobs for them complete. Unmatched reads are aggregated into unmatched_logfile.xlsx, keeping only the highest-count combinations per input file (the long tail of near-random spacer variants carries no diagnostic value).

License

MIT