emtax 0.2.5

Bioinformatics workflow management tool for taxonomic profiling on HPC systems

EM-TAX

A Python package for running taxonomic profiling workflows on HPC systems with conda environment support.

Overview

EM-TAX simplifies the process of running taxonomic profiling workflows on High-Performance Computing (HPC) systems. It automates the following tasks:

• Connecting to HPC systems using SSH
• Uploading raw sequencing data
• Downloading and setting up reference databases
• Configuring and submitting Snakemake workflows with conda environment support
• Monitoring job progress and checking job status
• Retrieving results and abundance matrices

Installation

pip install emtax

Prerequisites

• SSH access to an HPC system
• Python 3.9 or higher
• Conda or Mamba installed on the HPC system (for environment activation)

Conda Environment Setup

EMTAX requires a conda environment on the HPC system. While EMTAX can automatically generate the environment file, you may want to manually create it before running the workflow:

1. Create a file named environment.yaml with the following content (do NOT run these lines as commands!):

Tip: Save the following block as environment.yaml and create the environment with:

conda env create -f environment.yaml

name: emtax_env
channels:
  - bioconda
  - conda-forge
  - defaults
dependencies:
  - python=3.9.19
  - snakemake-minimal=7.32.4
  - kraken2=2.1.3
  - bracken=2.8
  - krona=2.7.1
  - fastp=0.23.4
  - bowtie2=2.5.2
  - samtools=1.18
  - pandas=2.1.1
  - numpy=1.23.5
  - biopython=1.81
  - scikit-bio=0.5.8
  - pip=23.3.1
  - pip:

2. Create the conda environment on the HPC system:

# SSH into your HPC system
ssh [e-mail]

# Navigate to your working directory
cd /path/to/your/project

# Create the conda environment from the YAML file
conda env create -f environment.yaml

# Activate the environment
conda activate emtax_env

3. Once the environment is created, you can submit your EMTAX job using the --conda-env option to specify the environment name:

emtax hpc \
  -r /path/to/sample1_L001_R1.fastq.gz \
  [...other options...] \
  --conda-env emtax_env

Usage

Basic Commands

Submit a job

For paired-end (with lane) data:

emtax hpc \
  -r /path/to/sample1_L001_R1.fastq.gz \
  -r /path/to/sample1_L001_R2.fastq.gz \
  -r /path/to/sample1_L002_R1.fastq.gz \
  -r /path/to/sample1_L002_R2.fastq.gz \
  -o /path/on/hpc/output \
  --kraken-db /path/on/hpc/kraken2_db \
  --corn-db /path/on/hpc/corn_db \
  --host hpc.example.edu \
  --username myuser \
  --password-auth

For paired-end (without lane) data:

emtax hpc \
  -r /path/to/sample1_R1.fastq.gz \
  -r /path/to/sample1_R2.fastq.gz \
  -o /path/on/hpc/output \
  --kraken-db /path/on/hpc/kraken2_db \
  --corn-db /path/on/hpc/corn_db \
  --host hpc.example.edu \
  --username myuser \
  --password-auth

For single-end data:

emtax hpc \
  -r /path/to/sample1.fastq.gz \
  -o /path/on/hpc/output \
  --kraken-db /path/on/hpc/kraken2_db \
  --corn-db /path/on/hpc/corn_db \
  --host hpc.example.edu \
  --username myuser \
  --password-auth

Example with real paths:

emtax hpc \
  -r /Users/username/Downloads/S1_L001_R1.fastq.gz \
  -r /Users/username/Downloads/S1_L001_R2.fastq.gz \
  -r /Users/username/Downloads/S1_L002_R1.fastq.gz \
  -r /Users/username/Downloads/S1_L002_R2.fastq.gz \
  -o /home/username/toxotest/output \
  --kraken-db /home/username/toxotest/Kraken2_DB \
  --corn-db /home/username/toxotest/Zm-B73-REFERENCE-NAM-5.0 \
  --host hpcr8o2rnp.uta.edu \
  --username username \
  --password-auth

Check job status

emtax status 12345 --host hpc.example.edu --username myuser --password-auth

Example with real job ID:

emtax status 23104 --host hpcr8o2rnp.uta.edu --username username --password-auth

Download results

emtax download \
  --output-dir /path/on/hpc/output \
  --local-dir ./results \
  --host hpc.example.edu \
  --username myuser

Download abundance matrix

To download just the abundance matrix file (the final result of the workflow):

emtax get-abundance \
  --output-dir /path/on/hpc/output \
  --local-dir ./results \
  --host hpc.example.edu \
  --username myuser \
  --password-auth

You can also wait for a specific job to complete before downloading:

emtax get-abundance \
  --output-dir /path/on/hpc/output \
  --local-dir ./results \
  --host hpc.example.edu \
  --username myuser \
  --password-auth \
  --job-id 23110

Command Options

HPC Command Options

• -r, --raw-files: Raw FASTQ files to process (use multiple -r flags for multiple files)
• -o, --output-dir: Output directory on the HPC system
• --kraken-db: Path to Kraken2 database on the HPC system
• --corn-db: Path to corn genome database for host removal
• --host: HPC hostname (default: uses SSH config)
• --username: HPC username (default: uses SSH config)
• --identity-file: Path to SSH identity file/private key (e.g., ~/.ssh/id_rsa_toxolib)
• --password-auth: Use password authentication instead of SSH keys
• --partition: HPC partition/queue to use (default: normal)
• --threads: Number of threads to request (default: 16)
• --memory: Memory to request (default: 200GB)
• --time: Time limit for the job (default: 48:00:00)
• --conda-env: Specify a custom conda environment name (default: emtax_env)
• --no-download-db: Skip downloading reference databases
• --no-upload-data: Skip uploading raw data files (use if files are already on HPC)
• --dry-run: Show what would be done without actually connecting to HPC
• --help: Show help message

Status Command Options

• job_id: The SLURM job ID to check (required)
• --host: HPC hostname
• --username: HPC username
• --identity-file: Path to SSH identity file (private key)
• --password-auth: Use password authentication instead of key-based

Download Command Options

• --output-dir: HPC output directory path (required)
• --local-dir: Local directory to save downloaded files (default: current directory)
• --host: HPC hostname
• --username: HPC username
• --identity-file: Path to SSH identity file (private key)
• --password-auth: Use password authentication instead of key-based
• --file-pattern: File pattern to download (default: *.csv)

Get-Abundance Command Options

• --output-dir: HPC output directory path (required)
• --local-dir: Local directory to save the abundance matrix file (default: current directory)
• --host: HPC hostname
• --username: HPC username
• --identity-file: Path to SSH identity file (private key)
• --password-auth: Use password authentication instead of key-based
• --job-id: Job ID to check before downloading (will wait for job completion)

Reference Databases

TaxoPipe can automatically download and set up the required reference databases:

• Kraken2 Database: Standard microbial database for taxonomic classification

  • Source: https://genome-idx.s3.amazonaws.com/kraken/k2_standard_20240112.tar.gz

• Corn Genome Database: Used for host removal

  • Source: https://glwasoilmetagenome.s3.us-east-1.amazonaws.com/corn_db.zip

Manual Database Setup

If you prefer to manually download and set up the reference databases:

1. Download and prepare the Kraken2 database:

# On your local machine
# Download the Kraken2 database
wget https://genome-idx.s3.amazonaws.com/kraken/k2_standard_20230605.tar.gz

# Extract the database
mkdir -p k2_standard
tar -xzvf k2_standard_20230605.tar.gz -C k2_standard

2. Download and prepare the Corn database:

# On your local machine
# Download the Corn database
wget https://genome-idx.s3.amazonaws.com/bt/Zm-B73-REFERENCE-NAM-5.0.zip

# Extract the database
unzip Zm-B73-REFERENCE-NAM-5.0.zip

3. Transfer databases to HPC:

# First, create the destination directories on HPC
ssh [e-mail] "mkdir -p /path/on/hpc/Kraken2_DB"
ssh [e-mail] "mkdir -p /path/on/hpc/Zm-B73-REFERENCE-NAM-5.0"

Option A: Using scp (simple but may fail with large directories)

# Transfer Kraken2 database
scp -r k2_standard/* [e-mail]:/path/on/hpc/Kraken2_DB/

# Transfer Corn database
scp -r Zm-B73-REFERENCE-NAM-5.0/* [e-mail]:/path/on/hpc/Zm-B73-REFERENCE-NAM-5.0/

Option B: Using rsync (recommended for large transfers)

# Transfer Kraken2 database
rsync -avz k2_standard/ [e-mail]:/path/on/hpc/Kraken2_DB/

# Transfer Corn database
rsync -avz Zm-B73-REFERENCE-NAM-5.0/ [e-mail]:/path/on/hpc/Zm-B73-REFERENCE-NAM-5.0/

Option C: Using tar+ssh (for troubleshooting transfer issues)

# For Kraken2 database
tar -czf - k2_standard | ssh [e-mail] "tar -xzf - -C /path/on/hpc && mv /path/on/hpc/k2_standard/* /path/on/hpc/Kraken2_DB/ && rmdir /path/on/hpc/k2_standard"

# For Corn database
tar -czf - Zm-B73-REFERENCE-NAM-5.0 | ssh [e-mail] "tar -xzf - -C /path/on/hpc && mv /path/on/hpc/Zm-B73-REFERENCE-NAM-5.0/* /path/on/hpc/Zm-B73-REFERENCE-NAM-5.0/ && rmdir /path/on/hpc/Zm-B73-REFERENCE-NAM-5.0"

Troubleshooting Transfer Issues:

If you encounter "path canonicalization failed" or other transfer errors:

1. Verify the destination directory exists:

   ssh [e-mail] "ls -la /path/on/hpc/"
   

2. Check permissions:

   ssh [e-mail] "ls -la /path/on/hpc/ | grep Kraken2_DB"
   ssh [e-mail] "ls -la /path/on/hpc/ | grep Zm-B73"
   

3. Try transferring files individually:

   # For Corn database
   for file in Zm-B73-REFERENCE-NAM-5.0/*.bt2; do
     scp "$file" [e-mail]:/path/on/hpc/Zm-B73-REFERENCE-NAM-5.0/
   done
   

Transfer your sequencing data:

# On your local machine
# Create a directory for your data on HPC
ssh [e-mail] "mkdir -p /path/on/hpc/output/Raw_Data"

# Transfer your FASTQ files
scp /path/to/local/sample_L001_R1.fastq.gz [e-mail]:/path/on/hpc/output/Raw_Data/
scp /path/to/local/sample_L001_R2.fastq.gz [e-mail]:/path/on/hpc/output/Raw_Data/
# Repeat for all your FASTQ files

Run EMTAX with the --no-upload-data flag:

emtax hpc \
  -r /path/to/sample1_L001_R1.fastq.gz \
  -r /path/to/sample1_L001_R2.fastq.gz \
  -o /path/on/hpc/output \
  --kraken-db /path/on/hpc/Kraken2_DB \
  --corn-db /path/on/hpc/Zm-B73-REFERENCE-NAM-5.0 \
  --host hpc.example.edu \
  --username myuser \
  --password-auth \
  --no-upload-data

Output Structure

output/
├── Raw_Data/                  # Raw input files
├── workflow/                  # Workflow files and scripts
│   ├── Snakefile              # Snakemake workflow definition
│   ├── config.yaml            # Workflow configuration
│   ├── environment.yaml       # Conda environment definition
│   ├── scripts/               # Helper scripts
│   │   └── create_abundance_matrix.py  # Script for abundance matrix creation
│   ├── submit_job.sh          # Job submission script
│   ├── emtax_[job_id].out  # SLURM output log file
│   └── emtax_[job_id].err  # SLURM error log file

Job Log Files

EMTAX generates log files for each job submission. These log files contain important information about the job execution, including any errors or warnings.

Log File Locations

The log files are created in the workflow directory with the following naming pattern:

• Output log: [output_dir]/workflow/emtax_[job_id].out
• Error log: [output_dir]/workflow/emtax_[job_id].err

For example, if your output directory is /home/username/toxotest/output and your job ID is 23109, the log files would be located at:

/home/username/toxotest/output/workflow/emtax_23109.out
/home/username/toxotest/output/workflow/emtax_23109.err

Viewing Log Files

You can view the log files using the following commands:

# SSH into your HPC system
ssh [e-mail]

# View the output log
cat /path/to/output/workflow/emtax_[job_id].out

# View the error log
cat /path/to/output/workflow/emtax_[job_id].err

# Or use 'tail' to follow the logs in real-time
tail -f /path/to/output/workflow/emtax_[job_id].out

Results Structure

Results/
└── Taxonomic_Profiling/
    ├── 1_DNA_Kraken2/                   # Kraken2 classification results
    ├── 2_DNA_Bracken/                   # Bracken abundance estimation
    └── 5_DNA_Relative_Abundance_Matrix_Python/  # Abundance matrices

Flexible Input File Support

EM-TAX now supports multiple input file formats and automatically detects the file type:

Supported Input Formats

1. Multi-lane paired-end files: Files with lane information (e.g., S1_L001_R1.fastq.gz, S1_L001_R2.fastq.gz)
2. Lane-combined paired-end files: Files without lane information (e.g., S1_R1.fastq.gz, S1_R2.fastq.gz)
3. Single-end files: Files without pair information (e.g., S1.fastq.gz)

The workflow automatically detects the file format and processes accordingly:

• For multi-lane files, it combines the lanes for each sample
• For lane-combined files, it uses them directly
• For single-end files, it processes them in single-end mode without creating unnecessary empty R2 files

Important Notes for Single-End Data

• When using true single-end data, provide files with the naming pattern SampleName.fastq.gz (without R1/R2 designation)
• Do not concatenate paired-end files (R1+R2) to create a single-end file, as this will lead to suboptimal results
• All tools in the workflow (fastp, bowtie2, kraken2) will automatically run in the appropriate mode based on the input file type

Workflow Features

• Automatic Authentication: Multiple authentication methods with fallback options
• Network Resilience: Handles network connectivity issues and provides offline mode support
• Conda Environment Support: Automatically activates conda environments on the HPC system
• Flexible Environment Management: Creates minimal conda environments with retry logic
• Progress Tracking: Detailed logging and progress bars for file transfers
• Job Status Checking: Monitors job status and provides detailed information
• Result Retrieval: Downloads abundance matrices and other result files
• Raw Data Management: Efficiently uploads raw data with duplicate detection
• Flexible Input Support: Automatically detects and processes single-end and paired-end files

License

MIT License