eDentity-metabarcoding-pipeline

• Overview
• Installation
• Usage
  • Using Command Line Arguments
  • Using a Configuration File
  • Per-Sample Metadata (Primers, Plate & Well)
  • QC-Only Mode and Plate Summary
  • Configuring Snakemake Parameters via Profile
  • Pipeline Output Directory Structure

Overview

eDentity is a Snakemake-based metabarcoding pipeline for processing Illumina and AVITI paired-end sequencing data. It takes raw FASTQ files and produces Exact Sequence Variants (ESVs) through an automated sequence of quality control, primer trimming, merging, filtering, dereplication, and denoising steps. The pipeline also supports a lightweight QC-only mode and generates interactive per-plate yield reports to assist with sequencing run quality assessment. The pipeline is inspired by APSCALE and uses Vsearch, FastP, Cutadapt, and MultiQC under the hood.

This project is mantained by the Data Competence Group at Naturalis Biodiversity Center, funded by eDentity: A Dutch national eDNA infrastructure.

Installation

Install edentity alongside its dependencies with the command below;

conda create -n edentity-env \
  python=3.12.8 \
  fastp=0.24.0 \
  cutadapt=4.9 \
  vsearch=2.28.1 \
  biopython=1.84 \
  multiqc=1.27.1 \
  nbitk=0.5.9 \
  edentity \
  polars=1.30.0 \
  -c conda-forge -c bioconda -y && \
  conda activate edentity-env

Usage

After installation, the pipeline can be run from the command line. Parameters can be provided either directly via command line arguments or through a configuration file.

Pipeline modes

In addition to the full pipeline run, the following flagged modes are available:

Using Command Line Arguments

Replace the example parameters with those specific to your project:

edentity --raw_data_dir /path/to/your/raw_fastq_files/ \
--work_dir /path/to/your/work_directory \
--forward_primer pcr primer sequence \
--reverse_primer pcr primer sequence \
--min_length 200 \
--max_length 600

Using a Configuration File

Create a params_config.yaml file and copy the YAML template below into it. Adjust the parameters to your project specifications:

# project specific
raw_data_dir:  # "/path/to/your/raw_fastq_files/"
work_dir:  # "path/to/your/work_directory"
make_json_reports: False
dataType: "Illumina" # [Illumina, AVITI], one of the two
cpu_cores: 20 

# general quality control (Fastp)
average_qual: 25
length_required: 100
n_base_limit: 0

# PE_merging (these are set to vsearch default values)
maxdiffpct: 100
maxdiffs: 10
minovlen: 10

# primer_trimming (cutadapt)
forward_primer:   
reverse_primer: 
anchoring: False
discard_untrimmed: True

# quality_filtering (vsearch)
min_length: 100
max_length: 600
maxEE: 1

# dereplication (vsearch)
fasta_width: 0

# denoising (vsearch)
alpha: 2
minsize: 4

Then run the pipeline with:

edentity --config_file params_config.yaml

Parameters:

• --forward_primer: Forward primer sequence (global, applies to all samples).
• --reverse_primer: Reverse primer sequence (global, applies to all samples).
• --metadata_tsv: Path to a TSV file with per-sample metadata — primers, plate ID, and well position (see Per-Sample Metadata).
• --raw_data_dir: Directory containing your raw sequencing data.
• --work_dir: Directory for pipeline outputs and intermediate files.
• --make_json_reports: Set true to create extended json reports

Per-Sample Metadata (Primers, Plate & Well)

A single --metadata_tsv file is used for all per-sample metadata: primer sequences, PCR plate IDs, and well positions. This is the recommended approach when a run contains samples with different primers, or when you want to generate plate yield reports.

Recommendation: Where possible, run each marker gene as a separate pipeline invocation with its own --work_dir. This keeps outputs cleanly separated and makes downstream analysis simpler. The metadata TSV is provided as a convenience for mixed runs.

TSV format

Create a tab-separated file. The sample column is always required; all other columns are optional and used only when relevant.

• sample must match sample names exactly as they appear in the raw FASTQ filenames.
• forward_primer / reverse_primer override the global --forward_primer / --reverse_primer per sample. Use comma-separated sequences for multiple primer pairs per sample.
• plate_id and well are required for per-plate grouping in QC mode and for --plate_summary_only.

Usage

edentity --raw_data_dir /path/to/raw_fastq/ \
--work_dir /path/to/work_dir \
--metadata_tsv /path/to/metadata.tsv

It can also be specified in a config file:

metadata_tsv: "/path/to/metadata.tsv"

QC-Only Mode and Plate Summary

QC-only mode

Run only the quality-control steps (FastP + paired-end merging) without executing the full metabarcoding workflow. A MultiQC report is generated at the end.

edentity --raw_data_dir /path/to/raw_fastq/ \
--work_dir /path/to/work_dir \
--forward_primer PRIMER_F \
--reverse_primer PRIMER_R \
--qc_only

Plate yield report in QC mode

--qc_only always produces a plate yield HTML report. For per-plate grouping (one section per plate with a 96-well grid), provide a --metadata_tsv with plate_id and well columns.

edentity --raw_data_dir /path/to/raw_fastq/ \
--work_dir /path/to/work_dir \
--metadata_tsv /path/to/metadata.tsv \
--qc_only

The plate yield report is written to:

work_dir/Results/report/{work_dir}_plate_yield.html

Standalone plate yield report

Generate (or regenerate) a plate yield report from an existing summary TSV without re-running the pipeline. A --metadata_tsv with plate_id and well columns is required.

edentity --work_dir /path/to/work_dir \
--plate_summary_only \
--metadata_tsv /path/to/metadata.tsv

Relevant flags:

Re-extracting JSON metadata

Re-generate the _metabarcoding.json output from a completed pipeline run without re-running any pipeline steps. Only --work_dir is required.

edentity --work_dir /path/to/work_dir --metadata_only

Configuring Snakemake Parameters via Profile

You can control Snakemake-specific parameters (such as cluster execution, resource limits, and rerun-incomplete ...) using a profile YAML configuration. This is useful for running the pipeline on HPC clusters or customizing workflow execution.

Create a snakemake-profile.yaml file with content like:

executor: local # clusters e.g slurm, lsf, aws-batch ... see snakemake documentation 
jobs: "30"
max-jobs-per-second: "10"
max-status-checks-per-second: "10"
local-cores: 44
latency-wait: "30"
printshellcmds: "True"
rerun-incomplete: "False"
keep-incomplete: "True"
conda-cleanup-envs: "False"
dryrun: true
resources:
    mem_mb: 16000
    threads: 8

• executor: Cluster scheduler (e.g., SLURM).
• jobs: Maximum number of parallel jobs.
• resources: Default resource limits for jobs.
• dryrun: Set to true to perform a dry-run (no jobs will be executed).

For more details on these and other Snakemake parameters, see the Snakemake documentation.

To use this profile, run:

edentity --profile snakemake-profile.yaml --config_file params_config.yaml

Snakemake parameters can also be provided directly via the command line, but they must be specified in their long form (e.g., --jobs instead of -j). Command-line parameters take precedence over those defined in the profile configuration file or the default parameters.

For example, you can use both a profile configuration file and override specific parameters via the command line:

edentity --profile snakemake-profile.yaml --config_file params_config.yaml \
--jobs 50 --latency-wait 60 --until merge 

In this example:

• The --config_file option specifies the parameters specific to eDentity, such as input directories, primers, and quality control settings.
• The --profile option specifies the Snakemake profile configuration file, which controls the behavior of Snakemake, such as job execution, resource limits, and cluster settings.
• The --jobs, --latency-wait, and --until parameters override the corresponding values in the profile configuration file.
• Command-line parameters always take priority over the profile or default settings.

For a full list of options params:

edentity --help

Pipeline Output Directory Structure

After successful execution, the pipeline generates a structured set of output directories and files within your specified work_dir. All file names are prefixed with your work_dir. The main components are:

Full pipeline run

work_dir/
├── Results/
│   ├── ESVs_fasta/                          # FASTA file of Exact Sequence Variants
│   └── report/
│       ├── {work_dir}_ESV_table.tsv         # ESV abundance table
│       ├── {work_dir}_summary_report.tsv    # Per-sample summary statistics
│       ├── {work_dir}_multiqc_reports/
│       │   └── {work_dir}_multiqc.html      # Interactive MultiQC report
│       └── fastpQC/                         # Per-sample FastP JSON reports
├── logs/                                    # Log files for each pipeline step
│   ├── fastpQC/
│   ├── merge/
│   ├── trimming/
│   ├── filter/
│   ├── dereplication/
│   ├── denoise/
│   ├── chimera/
│   ├── search_exact/
│   └── multiqc/
└── edentity_pipeline_settings/              # Configuration files used for this run
    ├── {work_dir}_snakemake_config.yml
    ├── {work_dir}_snakemake_profile/
    └── multiqc_config/

QC-only mode (--qc_only)

work_dir/
├── Results/
│   └── report/
│       ├── {work_dir}_summary_report.tsv    # Per-sample QC summary
│       └── {work_dir}_multiqc_reports/
│           └── {work_dir}_QC_only.html      # Interactive MultiQC QC report
├── logs/
│   ├── fastpQC/
│   ├── merge/
│   └── multiqc/
└── edentity_pipeline_settings/

QC-only + plate summary (--qc_only --plate_summary)

work_dir/
├── Results/
│   └── report/
│       ├── {work_dir}_summary_report.tsv
│       ├── {work_dir}_QC_only.html          # MultiQC QC report
│       └── {work_dir}_plate_yield.html      # Self-contained plate yield HTML report
├── logs/
│   ├── fastpQC/
│   ├── merge/
│   ├── multiqc/
│   └── plate_summary/
└── edentity_pipeline_settings/

Standalone plate yield (--plate_summary_only)

Reads {work_dir}_summary_report.tsv from an existing run and writes one new file:

work_dir/
└── Results/
    └── report/
        └── {work_dir}_plate_yield.html      # Self-contained plate yield HTML report