hostile 0.2.0

Accurate host read removal

Hostile

Hostile removes host sequences from short and long reads, consuming paired or unpaired fastq[.gz] input. Batteries are included – a human reference genome is downloaded when run for the first time. Hostile is precise by default, removing an order of magnitude fewer microbial reads than existing approaches while removing >99.5% of real human reads from 1000 Genomes Project samples. For ultimate precision, a prebuilt masked reference can be downloaded, or a new one created for chosen target organisms. Read headers can be replaced with integers (using --rename) for privacy and smaller FASTQs. Heavy lifting is done with fast existing tools (Minimap2/Bowtie2 and Samtools). Bowtie2 is the default aligner for short (paired) reads while Minimap2 is default aligner for long reads. Further information and benchmarks can be found in the BioRxiv preprint and this blog post. Feel free open an issue, tweet, toot or email me to report problems or suggest improvements.

Reference genomes & indexes

The default human-t2t-hla index is a good choice for most use cases, and is downloaded automatically when running Hostile for the first time. Slightly higher microbial retention in may be achieved by specifying a custom --index masked against target organisms. The human-t2t-hla-argos985 index is masked against 985 reference grade bacterial genomes, making it a good choice for decontaminating bacterial genomes. Another masked genome human-t2t-hla-argos985-mycob140 was created specifically for maximising the retention of mycobacterial genomes. Note that Bowtie2 indexes need to be untarred before use, while Minimap2 indexes are built at runtime from the specified genome in fasta[.gz] format. The index human-t2t-hla and human-t2t-hla-argos985-mycob140 were compared in the paper.

Name	Composition	Minimap2 genome	Bowtie2 index	Date
human-t2t-hla (default)	T2T-CHM13v2.0 + IPD-IMGT/HLA v3.51	human-t2t-hla.fa.gz	human-t2t-hla.tar	2023-07
human-t2t-hla-argos985	T2T-CHM13v2.0 & IPD-IMGT/HLA v3.51; masked with 985 FDA-ARGOS 150mers	human-t2t-hla-argos985.fa.gz	human-t2t-hla-argos985.tar	2023-07
human-t2t-hla-argos985-mycob140	T2T-CHM13v2.0 & IPD-IMGT/HLA v3.51; masked with 985 FDA-ARGOS & 140 mycobacterial 150mers	human-t2t-hla-argos985-mycob140.fa.gz	human-t2t-hla-argos985-mycob140.tar	2023-07

Install

Installation with conda/mamba or Docker is recommended due to non-Python dependencies (Bowtie2, Minimap2, Samtools and Bedtools). Hostile is tested with Ubuntu Linux 22.04, MacOS 12, and under WSL for Windows.

Conda/mamba

conda create -n hostile -c conda-forge -c bioconda hostile  # Mamba is faster
conda activate hostile

Docker

docker run quay.io/biocontainers/hostile:0.1.0--pyhdfd78af_0

# Build your own
wget https://raw.githubusercontent.com/bede/hostile/main/Dockerfile
docker build . --platform linux/amd64

Development install

git clone https://github.com/bede/hostile.git
cd hostile
conda env create -f environment.yml  # Mamba/Micromamba are faster
conda activate hostile
pip install --editable '.[dev]'
pytest

Command line usage

$ hostile clean --help
usage: hostile clean [-h] --fastq1 FASTQ1 [--fastq2 FASTQ2] [--aligner {bowtie2,minimap2,auto}] [--index INDEX] [--rename] [--out-dir OUT_DIR] [--threads THREADS] [--force] [--debug]

Remove host reads from paired fastq(.gz) files

options:
  -h, --help            show this help message and exit
  --fastq1 FASTQ1       path to forward fastq(.gz) file
  --fastq2 FASTQ2       optional path to reverse fastq(.gz) file
                        (default: None)
  --aligner {bowtie2,minimap2,auto}
                        alignment algorithm
                        (default: auto)
  --index INDEX         path to custom genome or index. For Bowtie2, provide an index path without the .bt2 extension
                        (default: None)
  --rename              replace read names with incrementing integers
                        (default: False)
  --out-dir OUT_DIR     path to output directory
                        (default: ./)
  --threads THREADS     number of CPU threads to use
                        (default: 10)
  --force               overwrite existing output files
                        (default: False)
  --debug               show debug messages
                        (default: False)

Short reads

$ hostile clean --fastq1 reads.r1.fastq.gz --fastq2 reads.r2.fastq.gz
INFO: Using Bowtie2 (paired reads)
INFO: Found cached index (/Users/bede/Library/Application Support/hostile/human-t2t-hla)
INFO: Cleaning…
INFO: Complete
[
    {
        "aligner": "bowtie2",
        "index": "/path/to/data/dir/human-t2t-hla",
        "fastq1_in_name": "reads.r1.fastq.gz",
        "fastq2_in_name": "reads.r2.fastq.gz",
        "fastq1_in_path": "/path/to/hostile/reads.r1.fastq.gz",
        "fastq2_in_path": "/path/to/hostile/reads.r2.fastq.gz",
        "fastq1_out_name": "reads.r1.clean_1.fastq.gz",
        "fastq2_out_name": "reads.r2.clean_2.fastq.gz",
        "fastq1_out_path": "/path/to/hostile/reads.r1.clean_1.fastq.gz",
        "fastq2_out_path": "/path/to/hostile/reads.r2.clean_2.fastq.gz",
        "reads_in": 20,
        "reads_out": 20,
        "reads_removed": 0,
        "reads_removed_proportion": 0.0
    }
]

$ hostile clean --rename --fastq1 reads_1.fastq.gz --fastq2 reads_2.fastq.gz \
  --index /path/to/human-t2t-hla-argos985-mycob140 > decontamination-log.json
INFO: Using Bowtie2
INFO: Found cached index (/Users/bede/Library/Application Support/hostile/human-t2t-hla)
INFO: Cleaning…
INFO: Complete

Long reads

$ hostile clean --fastq1 tests/data/h37rv_10.r1.fastq.gz
INFO: Using Minimap2's long read preset (map-ont)
INFO: Found cached index (/Users/bede/Library/Application Support/hostile/human-t2t-hla)
INFO: Cleaning…
INFO: Complete
[
    {
        "aligner": "minimap2",
        "index": "/Users/bede/Library/Application Support/hostile/human-t2t-hla.fa.gz",
        "fastq1_in_name": "reads.fastq.gz",
        "fastq1_in_path": "/path/to/hostile/reads.fastq.gz",
        "fastq1_out_name": "reads.clean.fastq.gz",
        "fastq1_out_path": "/path/to/hostile/reads.clean.fastq.gz",
        "reads_in": 10,
        "reads_out": 10,
        "reads_removed": 0,
        "reads_removed_proportion": 0.0
    }
]

Python usage

from pathlib import Path
from hostile.lib import clean_paired_fastqs, ALIGNER

# Long reads, defaults
clean_fastqs(
    fastqs=[Path("reads.fastq.gz")],
)

# Paired short reads, all the options, capture log
log = lib.clean_paired_fastqs(
    fastqs=[(Path("reads_1.fastq.gz"), Path("reads_2.fastq.gz"))],
    aligner=ALIGNER.minimap2,
    index=Path("reference.fasta.gz"),
    out_dir=Path("decontaminated-reads"),
    force=True,
    threads=4
)

print(log)

Masking reference genomes

The mask subcommand makes it easy to create custom-masked reference genomes and achieve maximum retention of specific target organisms:

hostile mask human.fasta lots-of-bacterial-genomes.fasta --threads 8

You may wish to use one of the existing reference genomes as a starting point. Masking uses Minimap2's asm10 preset to align the supplied target genomes with the reference genome, and bedtools to mask out all aligned regions. For Bowtie2—the default aligner for decontaminating short reads—you will also need to build an index before you can use your masked genome with Hostile.

bowtie2-build masked.fasta masked-index
hostile clean --index masked-index --fastq1 reads_1.fastq.gz --fastq2 reads_2.fastq.gz

Citation

@article {Constantinides2023,
	author = {Bede Constantinides and Martin Hunt and Derrick W Crook},
	title = {Hostile: accurate host decontamination of microbial sequences},
	elocation-id = {2023.07.04.547735},
	year = {2023},
	doi = {10.1101/2023.07.04.547735},
	publisher = {Cold Spring Harbor Laboratory},
	URL = {https://www.biorxiv.org/content/early/2023/07/21/2023.07.04.547735},
	eprint = {https://www.biorxiv.org/content/early/2023/07/21/2023.07.04.547735.full.pdf},
	journal = {bioRxiv}
}