Usage

Help:

usage: fastq2bcl [-h] [--version] [-v] [-vv] [-m MASK] -r1 R1 [-r2 R2] [-i1 I1] [-i2 I2] [-o OUTDIR] [--exclude-umi] [--exclude-index]

Convert fastq.gz reads and metadata in a bcl2fastq-able run directory

options:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  -v, --verbose         set loglevel to INFO
  -vv, --very-verbose   set loglevel to DEBUG
  -m MASK, --mask MASK  define mask in format 110N10Y10Y110N
  -r1 R1, --read-1 R1   fastq.gz with R1 reads
  -r2 R2, --read-2 R2   fastq.gz with R2 reads (optional)
  -i1 I1, --index-1 I1  fastq.gz with I1 reads (optional)
  -i2 I2, --index-2 I2  fastq.gz with I2 reads (optional)
  -o OUTDIR, --outdir OUTDIR
                        Set the output directory for mocked run. default: cwd
  --exclude-umi         Do not write UMI from the R1 and R2 fastq reads to the cycles
  --exclude-index       Do not write Index from the R1 and R2 fastq reads to the cycles

Usage examples:

fastq2bcl -r1 single.fastq.gz
fastq2bcl -r1 R1.fastq.gz -r2 R2.fastq.gz -i1 I1.fastq.gz -i2 I2.fastq.gz
fastq2bcl -o output_dir -r1 single.fastq.gz
fastq2bcl -o output_dir --exclude-index -r1 single.fastq.gz
fastq2bcl -o output_dir -m 100Y20N -r1 R1.fastq.gz -r2 R2.fastq.gz -i1 I1.fastq.gz -i2 I2.fastq.gz

Custom mask

By default fastq2bcl will generate a RunInfo.xml file where Reads entries are generated using the sequence length of fastq.gz files.

For exammple, if I give as input 2 pairs with length 300 bp and 2 indexes with length 8p the resulting RunInfo will be:

<?xml version="1.0"?>
<RunInfo xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Version="2">
    <Run Id="YYMMDD_run_0001_ABCD" Number="1">
        <Flowcell>ABCD</Flowcell>
        <Instrument>run</Instrument>
        <Date>YYMMDD</Date>
        <Reads>
            <Read c="300" Number="1" IsIndexedRead="N" />
            <Read NumCycles="8" Number="2" IsIndexedRead="Y" />
            <Read NumCycles="8" Number="3" IsIndexedRead="Y" />
            <Read NumCycles="300" Number="4" IsIndexedRead="N" />
        </Reads>
        <FlowcellLayout LaneCount="1" SurfaceCount="1" SwathCount="1" TileCount="1" />
    </Run>
</RunInfo>

You can provide a custom mask (string). For example for 1 pair 350 bp with 1 index of 8bp:

350N8Y

Install

use pip to install in edit mode:

pip install -e .

Install packages for dev in a mamba environment:

mamba create -n fastq2bcl
mamba install -n fastq2bcl -c conda-forge tox pyscaffold biopython pytest-cov

Scripts

In the directory scripts there are some useful tools:

• scripts/bcl2fastq_docker.sh run bcl2fastq with docker on the current directory. Run it inside a run directory.

• scripts/build_flowcells.sh generate all the test flowcells using the datasets in data/test directory

Test

use tox or pytest to test:

tox
pytest

To test with pytest you need also pytest-cov in your environment.

Lint

you can lint code with:

tox -e lint

Pre commit hook is already configured and can be installed with this command:

pre-commit install

Fastq sequence description

Fields in fastq description:

See also https://support.illumina.com/help/BaseSpace_OLH_009008/Content/Source/Informatics/BS/FileFormat_FASTQ-files_swBS.htm

Filter file

The filter files can be found in the BaseCalls directory. The filter file specifies whether a cluster passed filters. Filter files are generated at cycle 26 using 25 cycles of data. For each tile, one filter file is generated. Location: Data/Intensities/BaseCalls/L001 File format: s_[lane]_[tile].filter

The format is described below

Filter bytes example:

bytes([0, 0, 0, 0]) # prefix 0
bytes([3, 0, 0, 0]) # version 3
struct.pack("<I", cluster_count) # number of cluster in little endian unsigned int
bytes([1]*cluster_count) # For each cluster an unsigned 8-bits integer Where Bit 0 is pass or failed filter

1 == PASS FILTER
0 == NO PASS FILTER

In hexdump:

BYTES 0-3      BYTES 4-7      BYTES 8-11     BYTES 12-14
00 00 00 00    03 00 00 00    03 00 00 00    01 01 01

At bytes 8-11 I have 3 clusters and each cluster is represented by a an unsigned 8-bit integer.

Control file

The control files are binary files containing control results.

Locations file

The BCL to FASTQ converter can use different types of position files and will expect a type based on the version of RTA used The locs files can be found in the Intensities/L<lane> directories

Bcl file

The BCL files can be found in the BaseCalls directory inside the run directory: Data/Intensities/BaseCalls/L<lane>/C<cycle>.1

They are named as follows:

s_<lane>_<tile>.bcl

Format:

Stat file

The stats files can be found in the BaseCalls directory inside the run directory: Data/Intensities/BaseCalls/L00<lane>/C<cycle>.1

They are named as follows:

s_<lane>_<tile>.stats

The Stats file is a binary file containing base calling statistics; the content is described below.

The data is for clusters passing filter only:

References

• bcl2fastq source code from illumina downloads https://support.illumina.com/sequencing/sequencing_software/bcl2fastq-conversion-software/downloads.html

• Spec file from illumina support https://support.illumina.com/content/dam/illumina-support/documents/documentation/software_documentation/bcl2fastq/bcl2fastq_letterbooklet_15038058brpmi.pdf

• http://support-docs.illumina.com/IN/NovaSeq6000Dx_HTML/Content/IN/NovaSeq/SequencingOutputFiles_fNV.htm

• https://support.illumina.com/help/BaseSpace_OLH_009008/Content/Source/Informatics/BS/FileFormat_FASTQ-files_swBS.htm

• https://docs.python.org/3/library/struct.html#format-characters

See also mkdata.sh file in bcl2fastq source code for insights on bcl format.

Acknowledgments

• https://github.com/sottorivalab

• https://humantechnopole.it

Notes

This project is inspired by the test script https://github.com/ShawHahnLab/igseq/blob/dev/tools/fastq2bcl.py from https://github.com/ShawHahnLab

This project has been set up using PyScaffold 4.5. For details and usage information on PyScaffold see https://pyscaffold.org/.