A Lexogen tool for demultiplexing and index error correcting fastq files. Works with Lexogen i7, i5 and i1 barcodes.
Project description
idemux - inline barcode demultiplexing
Idemux is a command line tool designed to demultiplex paired-end FASTQ files from QuantSeq-Pool.
Idemux can demultiplex based on i7, i5, and i1 inline barcodes. While this tool can generally be used to demultiplex any barcodes (as long as they are correctly supplied and in the fastq header), it performs best when used in combination with Lexogen indices, as it will correct common sequencing errors in the sequenced barcodes. This will allow you to retain more reads from your sequencing experiment while minimizing cross contamination.
Idemux use is permitted under the following licence.
General usage:
idemux [-h] --r1 READ1 --r2 READ2 [--sample-sheet SAMPLE_SHEET]
--out OUTPUT_DIR [--i1-start I1_START] [--i5-rc] [-v]
Run idemux:
idemux --r1 read_1.fastq.gz --r2 read_2.fastq.gz --sample-sheet samples.csv --out /some/output/path --i1-start pos_in_read_2
Features
FASTQ file demultiplexing based on i7, i5, and i1 barcodes.
Correction of barcode sequencing errors to maximize read yield (only works with Lexogen 12 nt UDIs that have been sequenced at least 8 nt).
Reverse complementation in case the i5 index has been sequenced as reverse complement.
Getting started
To get stated with demultiplexing you need to:
1. Installation
Idemux is available on pypi. To install idemux via pip:
$ pip install idemux
If you want to install idemux into a virtual env (always a good idea to avoid dependency conflicts), do the following:
$ cd /path/you/want/it/installed/to # creates the venv $ virtualenv idemux # activates the venv, run 'deactivate' to deactivate $ source idemux/bin/activate $ pip install idemux
Alternatively, you can clone this repository and install from there:
$ cd /path/you/want/it/installed/to # creates the venv $ virtualenv idemux $ git clone https://github.com/Lexogen-Tools/idemux.git $ source idemux/bin/activate $ python setup.py install
Additionally, Idemux will soon be available via bioconda!
2. Preparing the sample sheet
In order to run idemux on your QuantSeq-Pool data, you first need to prepare a csv file. We call this csv a sample sheet and it specifies which barcodes correspond to each sample.
This is a necessity as the software needs to know which bins reads should be sorted into during demultiplexing. A sample sheet can easily be generated by filling in an excel spreadsheet and exporting it as csv.
Example sample sheet (i7, i5, and i1 demuliplexing):
sample_name,i7,i5,i1 sample_0,AAAACATGCGTT,CCCCACTGAGTT,AAAACATGCGTT sample_1,AAAATCCCAGTT,CCCCTAAACGTT,AAAATCCCAGTT sample_2,GAAAATTTACGC,GCCCCTTTCAGA,GAAAATTTACGC sample_3,AAACTAACTGTC,CCCATCCATGTA,AAACTAACTGTC
A sample sheet consists of 4 columns and always starts with the header illustrated above. ‘Sample_name’ values will be used as output file names, while the sequences specified in i7, i5, and i1 will be used for demultiplexing.
Therefore, only unique, specific combinations of sample names and barcodes are allowed. This means using duplicated or ambiguous combinations will result in an error. However, idemux will do its best to tell you where the problem lies, if this happens.
In brief, the rules are:
Sample names need to be unique.
Barcode combinations need to be unique.
i7 and/or i5 indices have to be used consistently within the csv file. i7 and/or i5 indices need to either be present for all samples or for none at all.
In contrast to i7/i5 indices, i1 indices can be used for a subset of samples in the csv file.
Absence of a barcode needs to be indicated by an empty field (no value between commas ,,).
If your i5 has been sequenced as reverse complement, do not enter the reverse complement sequences in the sample sheet. Use the --i5-rc option!
See below for more showcases of sample/barcode combinations that are allowed or not allowed.
3. Running idemux
Once you have installed the tool, you can run it by typing idemux in the terminal.
Idemux accepts the following arguments:
required arguments:
--r1 READ1 path to gzipped read 1 FASTQ file
--r2 READ2 path to gzipped read 2 FASTQ file
--sample-sheet CSV csv file describing sample names, and barcode combinations
--out OUTPUT_DIR where to write the output files
optional arguments:
--i5-rc when the i5 barcode has been sequenced as reverse complement.
make sure to always use non-reverse complement sequences in the sample sheet
--i1_start POS start position of the i1 index (1-based) on read 2 (default: 11)
-v, --version show program's version number and exit
-h, --help show help message and exit
Example commands:
# demultiplexes read 1 and 2 into the folder 'demux' idemux --r1 read_1.fastq.gz --r2 read_2.fastq.gz --sample-sheet samples.csv --out demux # demultiplexing assuming the i1 barcode starts at the first base idemux --r1 read_1.fastq.gz --r2 read_2.fastq.gz --sample-sheet samples.csv --out demux --i1_start 1 # demultiplexing assuming i5 is present as reverse complement in the fastq header # if the i5 has been sequenced as reverse complement use this option and provide # the NON reverse complement sequences in the sample sheet. idemux --r1 read_1.fastq.gz --r2 read_2.fastq.gz --sample-sheet samples.csv --out demux
After a successfully completed run, idemux will write a summary report to the output folder (‘demultipexing_stats.tsv’).
Technicalities
When you run idemux, the following will happen:
It will check if your sample sheet is okay. See here for examples.
It will check the FASTQ header for barcodes and it expects them in the following format:
single index (i7 or i5): @NB502007:379:HM7H2BGXF:1:11101:24585:1069 1:N:0:TCAGGTAANNTT
where TCAGGTAANNTT is the sequence of the i7 or i5 index
dual index (i7 and i5): @NB502007:379:HM7H2BGXF:1:11101:24585:1069 1:N:0:TCAGGTAANNTT+NANGGNNCNNNN
where TCAGGTAANNTT is the sequence of the i7 index and NANGGNNCNNNN is the sequence of the i5 index.
Reads with incorrect i7,i5 or i1 index sequences which can be corrected by idemux will be written to the correct output file. However, the incorrect index sequence will not be replaced in the read header. This allows for additional processing of the incorrect sequences.
Reads that cannot be demultiplexed will be written to undetermined_R{1/2}.fastq.gz.
When you demultiplex based on i1 inline barcodes, a successfully recognized barcode sequence of 12 nt will be cut out and removed from read 2. This will leave you with the 10 nt UMI + the nucleotides that potentially follow the i1 barcode.
This allows you to:
Use other software, such as UMI_tools, to deal with the 10nt UMI, if desired.
To demuliplex lanes where QuantSeq-Pool has been pooled with other libraries and read 2 has been sequenced longer than the actual barcode.
Help
If you are demuliplexing a large number of samples (more than 500), you might encounter the following error:
OSError: [Errno 24] Too many open files
This error occurs because most OS have a limit on how many files can be opened and written to at the same time. In order to temporarily increase the limit on Linux run:
# multiply your sample number*2 (as data is paired end) # then round to the next multiple of 1024 $ ulimit -n the_number_above
If you are looking for a permanent solution, you can change your ulimit values this way.
In case you experience any issues with this software please open an issue describing your problem. Make sure to post the version of the tool you are running (-v, --version) and your os.
Sample sheet examples
This is allowed:
# demultiplexing via full i7, i5, i1 sample_name,i7,i5,i1 sample_0,AAAACATGCGTT,CCCCACTGAGTT,AAAACATGCGTT sample_1,AAAATCCCAGTT,CCCCTAAACGTT,AAAATCCCAGTT # demultiplexing via full i7, i5 and sparse i1 sample_name,i7,i5,i1 sample_0,AAAACATGCGTT,CCCCACTGAGTT,AAAACATGCGTT sample_1,AAAATCCCAGTT,CCCCTAAACGTT, # demultiplexing via full i7, i5 sample_name,i7,i5,i1 sample_0,AAAACATGCGTT,CCCCACTGAGTT, sample_1,AAAATCCCAGTT,CCCCTAAACGTT, # demultiplexing via full i7, no i5 and sparse i1 sample_name,i7,i5,i1 sample_0,AAAACATGCGTT,,AAAACATGCGTT sample_1,AAAATCCCAGTT,, # demultiplexing via full i7 only sample_name,i7,i5,i1 sample_0,AAAACATGCGTT,, sample_1,AAAATCCCAGTT,, # demultiplexing via full i5 and i1 sample_name,i7,i5,i1 sample_0,,CCCCACTGAGTT,AAAACATGCGTT sample_1,,CCCCTAAACGTT,AAAATCCCAGTT # demultiplexing via full i5 and sparse i1 sample_name,i7,i5,i1 sample_0,,CCCCACTGAGTT,AAAACATGCGTT sample_1,,CCCCTAAACGTT, # demultiplexing via full i5 sample_name,i7,i5,i1 sample_0,,CCCCACTGAGTT, sample_1,,CCCCTAAACGTT, # demultiplexing via full i1 sample_name,i7,i5,i1 sample_0,,,AAAACATGCGTT sample_1,,,AAAATCCCAGTT
This is not allowed:
# missing i1 column (or any other) sample_name,i7,i5, sample_0,AAAACATGCGTT,CCCCACTGAGTT sample_1,AAAATCCCAGTT,CCCCTAAACGTT # duplicated barcode combination sample_name,i7,i5,i1 sample_0,AAAACATGCGTT,CCCCACTGAGTT,AAAACATGCGTT sample_1,AAAACATGCGTT,CCCCACTGAGTT,AAAACATGCGTT # duplicated sample names sample_name,i7,i5,i1 sample_0,AAAACATGCGTT,CCCCACTGAGTT,AAAACATGCGTT sample_0,AAAATCCCAGTT,CCCCTAAACGTT,AAAATCCCAGTT # mixed, potentially ambiguous indexing (full i7 and sparse i5, i1) sample_name,i7,i5,i1 sample_0,AAAACATGCGTT,CCCCACTGAGTT,AAAACATGCGTT sample_1,AAAATCCCAGTT,,AAAATCCCAGTT sample_2,GAAAATTTACGC,GCCCCTTTCAGA,GAAAATTTACGC sample_3,AAACTAACTGTC,,AAACTAACTGTC # mixed, potentially ambiguous indexing indexing (no i7, sparse i5 & i1) sample_name,i7,i5,i1 sample_0,,CCCCACTGAGTT, sample_1,,,AAAATCCCAGTT # mixed, potentially ambiguous indexing indexing (sparse i7, full i5 & i1) sample_name,i7,i5,i1 sample_0,,CCCCACTGAGTT,AAAACATGCGTT sample_1,AAAATCCCAGTT,CCCCTAAACGTT,AAAATCCCAGTT sample_2,,GCCCCTTTCAGA,GAAAATTTACGC sample_3,AAACTAACTGTC,CCCATCCATGTA,AAACTAACTGTC # missing comma separator sample_name,i7,i5,i1 sample_0,AAAACATGCGTTCCCCACTGAGTT,AAAACATGCGTT # no barcodes sample_name,i7,i5,i1 sample_0,,, # wrong column headers wrong_col_name,i7,i5,i1 sample_0,AAAACATGCGTT,CCCCACTGAGTT,AAAACATGCGTT
History
0.1.2 (2020-08-21)
bumped version number
0.1.1 (2020-08-21)
fixed rst files with linter
First release on test PyPI.
0.1.0 (2020-08-21)
First building version.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
