edentity 1.4.8

A Snakemake-based pipeline for amplicon processing

eDentity-metabarcoding-pipeline

• Overview
• Installation
  • Using pip
  • Using conda
• Usage
  • Using Command Line Arguments
  • Using a Configuration File
  • Configuring Snakemake Parameters via Profile
  • Pipeline Output Directory Structure

Overview

eDentity is a Snakemake based metabarcoding workflow designed for Illumina/AVITI paired-end data. It automates Vsearch commands to denoise paired-end Fastq sequences and generate Exact Sequence Variants (ESVs). The pipeline is inspired by APSCALE; please cite them if you use this pipeline.

Installation

Copy the dependencies below into a file (e.g., edentity-env.yaml), then create and activate the environment with:

priority: strict
name: edentity-env
channels:
    - conda-forge
    - bioconda
    - nodefaults
dependencies:
    - snakemake
    - pip
    - cutadapt=4.9
    - biopython=1.84
    - fastp=0.24.0
    - multiqc=1.27.1
    - vsearch=2.28.1
    - pip:
        - edentity

Install:

conda env create -f edentity-env.yaml --name edentity-env  && conda activate edentity-env

Usage

After installation, the pipeline can be run from the command line. Parameters can be provided either directly via command line arguments or through a configuration file.

Using Command Line Arguments

Replace the example parameters with those specific to your project:

edentity --raw_data_dir /path/to/your/raw_fastq_files/ \
--work_dir /path/to/your/work_directory \
--forward_primer pcr primer sequence \
--reverse_primer pcr primer sequence \
--min_length 200 \
--max_length 600

Using a Configuration File

Create a params_config.yaml file and copy the YAML template below into it. Adjust the parameters to your project specifications:

# project specific
raw_data_dir: "/path/to/your/raw_fastq_files/"
work_dir: "path/to/your/work_directory"
make_json_reports: False
dataType: "Illumina" # [Illumina, AVITI], one of the two
cpu_cores: 20 

# general quality control (Fastp)
average_qual: 25
length_required: 100
n_base_limit: 0

# PE_merging (these are set to vsearch default values)
maxdiffpct: 100
maxdiffs: 10
minovlen: 10

# primer_trimming (cutadapt)
forward_primer:   
reverse_primer: 
anchoring: False
discard_untrimmed: True

# quality_filtering (vsearch)
min_length: 100
max_length: 600
maxEE: 1

# dereplication (vsearch)
fasta_width: 0

# denoising (vsearch)
alpha: 2
minsize: 4

Then run the pipeline with:

edentity --config_file params_config.yaml

Parameters:

• --forward_primer: Forward primer sequence.
• --reverse_primer: Reverse primer sequence.
• --raw_data_dir: Directory containing your raw sequencing data.
• --work_dir: Directory for pipeline outputs and intermediate files.
• --make_json_reports: Set true to create extended json reports

Configuring Snakemake Parameters via Profile

You can control Snakemake-specific parameters (such as cluster execution, resource limits, and rerun-incomplete ...) using a profile YAML configuration. This is useful for running the pipeline on HPC clusters or customizing workflow execution.

Create a snakemake-profile.yaml file with content like:

executor: local # clusters e.g slurm, lsf, aws-batch ... see snakemake documentation 
jobs: "30"
max-jobs-per-second: "10"
max-status-checks-per-second: "10"
local-cores: 44
latency-wait: "30"
printshellcmds: "True"
rerun-incomplete: "False"
keep-incomplete: "True"
conda-cleanup-envs: "False"
dryrun: true
resources:
    mem_mb: 16000
    threads: 8

• executor: Cluster scheduler (e.g., SLURM).
• jobs: Maximum number of parallel jobs.
• resources: Default resource limits for jobs.
• dryrun: Set to true to perform a dry-run (no jobs will be executed).

For more details on these and other Snakemake parameters, see the Snakemake documentation.

To use this profile, run:

edentity --profile snakemake-profile.yaml --config_file params_config.yaml

You can combine this with your pipeline configuration file for full control over both workflow and execution parameters.

For a full list of options params:

edentity --help

Pipeline Output Directory Structure

After successful execution, the pipeline generates a structured set of output directories and files within your specified work_dir. All file names are prefixed with your work_dir. The main components are:

work_dir/
│   ├── Results/
│   │   ├── ESVs_fasta/                  # Directory containing FASTA file of ESVs
│   │   └── reports/                     # Reports generated by the pipeline
│   │       ├── ESV_table.tsv            # Table of Exact Sequence Variants (ESVs)
│   │       ├── summary_report.tsv       # Summary statistics for the run
│   │       ├── metabarcoding_run.json     # JSON report with run metadata and parameters
│   │       └── multiqc_report/            # Directory containing MultiQC output
│   │           └── multiqc.html           # Interactive MultiQC report
├── logs/                          # log files for each step of the pipeline
├── edentity_pipeline_settings/    # Stores configuration files used for the pipeline run