Skip to main content

eDentity-metabarcoding-pipeline

Overview

eDentity is a Snakemake-based metabarcoding pipeline for processing Illumina and AVITI paired-end sequencing data. It takes raw FASTQ files and produces Exact Sequence Variants (ESVs) through an automated sequence of quality control, primer trimming, merging, filtering, dereplication, and denoising steps. The pipeline also supports a lightweight QC-only mode and generates interactive per-plate yield reports to assist with sequencing run quality assessment. The pipeline is inspired by APSCALE and uses Vsearch, FastP, Cutadapt, and MultiQC under the hood.

This project is mantained by the Data Competence Group at Naturalis Biodiversity Center, funded by eDentity: A Dutch national eDNA infrastructure.

Installation

Install edentity alongside its dependencies with the command below;

conda create -n edentity-env \
  python=3.12.8 \
  fastp=0.24.0 \
  cutadapt=4.9 \
  vsearch=2.28.1 \
  biopython=1.84 \
  multiqc=1.27.1 \
  nbitk=0.5.9 \
  edentity \
  polars=1.30.0 \
  -c conda-forge -c bioconda -y && \
  conda activate edentity-env

Usage

After installation, the pipeline can be run from the command line. Parameters can be provided either directly via command line arguments or through a configuration file.

Pipeline modes

In addition to the full pipeline run, the following flagged modes are available:

Mode Flags When to use
QC only --qc_only Run FastP + paired-end merging and generate a MultiQC report with plate yield summary; skip denoising and downstream steps. Per-plate grouping requires --metadata_tsv with plate_id and well
Standalone plate yield --plate_summary_only Re-generate the plate yield HTML from a completed run; requires only --work_dir and --metadata_tsv
Metadata only --metadata_only Re-extract the _metabarcoding.json from a completed run without re-running any pipeline steps; requires only --work_dir

Using Command Line Arguments

Replace the example parameters with those specific to your project:

edentity --raw_data_dir /path/to/your/raw_fastq_files/ \
--work_dir /path/to/your/work_directory \
--forward_primer pcr primer sequence \
--reverse_primer pcr primer sequence \
--min_length 200 \
--max_length 600

Using a Configuration File

Create a params_config.yaml file and copy the YAML template below into it. Adjust the parameters to your project specifications:

# project specific
raw_data_dir:  # "/path/to/your/raw_fastq_files/"
work_dir:  # "path/to/your/work_directory"
make_json_reports: False
dataType: "Illumina" # [Illumina, AVITI], one of the two
cpu_cores: 20 

# general quality control (Fastp)
average_qual: 25
length_required: 100
n_base_limit: 0

# PE_merging (these are set to vsearch default values)
maxdiffpct: 100
maxdiffs: 10
minovlen: 10

# primer_trimming (cutadapt)
forward_primer:   
reverse_primer: 
anchoring: False
discard_untrimmed: True

# quality_filtering (vsearch)
min_length: 100
max_length: 600
maxEE: 1

# dereplication (vsearch)
fasta_width: 0

# denoising (vsearch)
alpha: 2
minsize: 4

Then run the pipeline with:

edentity --config_file params_config.yaml

Parameters:

  • --forward_primer: Forward primer sequence (global, applies to all samples).
  • --reverse_primer: Reverse primer sequence (global, applies to all samples).
  • --metadata_tsv: Path to a TSV file with per-sample metadata — primers, plate ID, and well position (see Per-Sample Metadata).
  • --raw_data_dir: Directory containing your raw sequencing data.
  • --work_dir: Directory for pipeline outputs and intermediate files.
  • --make_json_reports: Set true to create extended json reports

Per-Sample Metadata (Primers, Plate & Well)

A single --metadata_tsv file is used for all per-sample metadata: primer sequences, PCR plate IDs, and well positions. This is the recommended approach when a run contains samples with different primers, or when you want to generate plate yield reports.

Recommendation: Where possible, run each marker gene as a separate pipeline invocation with its own --work_dir. This keeps outputs cleanly separated and makes downstream analysis simpler. The metadata TSV is provided as a convenience for mixed runs.

TSV format

Create a tab-separated file. The sample column is always required; all other columns are optional and used only when relevant.

sample forward_primer reverse_primer plate_id well
Sample1 CTTGGTCATTTAGAGGAAGTAA GCTGCGTTCTTCATCGATGC CL288 A01
Sample2 GTGYCAGCMGCCGCGGTAA GGACTACNVGGGTWTCTAAT CL288 B01
Sample3 CTTGGTCATTTAGAGGAAGTAA;GCATCGATGAAGAACGCAGC GCTGCGTTCTTCATCGATGC;TCCTCCGCTTATTGATATGC CL288 C01
  • sample must match sample names exactly as they appear in the raw FASTQ filenames.
  • forward_primer / reverse_primer override the global --forward_primer / --reverse_primer per sample. Use semicolon-separated sequences for multiple primer pairs per sample.
  • plate_id and well are required for per-plate grouping in QC mode and for --plate_summary_only.

Usage

edentity --raw_data_dir /path/to/raw_fastq/ \
--work_dir /path/to/work_dir \
--metadata_tsv /path/to/metadata.tsv

It can also be specified in a config file:

metadata_tsv: "/path/to/metadata.tsv"

QC-Only Mode and Plate Summary

QC-only mode

Run only the quality-control steps (FastP + paired-end merging) without executing the full metabarcoding workflow. A MultiQC report is generated at the end.

edentity --raw_data_dir /path/to/raw_fastq/ \
--work_dir /path/to/work_dir \
--forward_primer PRIMER_F \
--reverse_primer PRIMER_R \
--qc_only

Plate yield report in QC mode

--qc_only always produces a plate yield HTML report. For per-plate grouping (one section per plate with a 96-well grid), provide a --metadata_tsv with plate_id and well columns.

edentity --raw_data_dir /path/to/raw_fastq/ \
--work_dir /path/to/work_dir \
--metadata_tsv /path/to/metadata.tsv \
--qc_only

The plate yield report is written to:

work_dir/Results/report/{work_dir}_plate_yield.html

Standalone plate yield report

Generate (or regenerate) a plate yield report from an existing summary TSV without re-running the pipeline. A --metadata_tsv with plate_id and well columns is required.

edentity --work_dir /path/to/work_dir \
--plate_summary_only \
--metadata_tsv /path/to/metadata.tsv

Relevant flags:

Flag Description
--qc_only Run QC steps only; always includes a plate yield report. Per-plate grouping requires --metadata_tsv with plate_id and well
--plate_summary_only Generate a plate yield report from an existing run's work_dir; no pipeline steps are run
--metadata_only Re-extract JSON metadata from a completed run without re-running the pipeline
--metadata_tsv TSV with per-sample metadata; required for plate reports — must include plate_id and well columns
--average_qual Minimum average quality score used by FastP (default: 25); reflected in the plate report footnote
--n_base_limit Maximum number of N bases per read (default: 0); reflected in the plate report footnote
--length_required Minimum read length after trimming (default: 100); reflected in the plate report footnote

Reading the plate yield report

Each plate page shows a 96-well grid. Hover over a well to see its sample, total reads, merged %, % with primers and ESV count (each shown only when the summary report has that column).

Well colour = read yield QC. A sample fails if it has fewer reads than 10% of the run's median total_reads. Samples that clear this cut-off but still have fewer than 10,000 reads are flagged as warn instead of pass:

Colour Status Meaning
Green pass at or above the 10%-of-median cut-off and ≥ 10,000 reads
Orange warn at or above the 10%-of-median cut-off but < 10,000 reads
Red fail below 10% of the run's median total_reads
Grey empty no sample assigned to the well

Diagonal stripes = no ESVs. A well whose esv_count is 0 is striped on top of its QC colour, so a red striped well failed QC and produced no ESVs, while a green striped well passed QC but still produced none.

Re-extracting JSON metadata

Re-generate the _metabarcoding.json output from a completed pipeline run without re-running any pipeline steps. Only --work_dir is required.

edentity --work_dir /path/to/work_dir --metadata_only

Configuring Snakemake Parameters via Profile

You can control Snakemake-specific parameters (such as cluster execution, resource limits, and rerun-incomplete ...) using a profile YAML configuration. This is useful for running the pipeline on HPC clusters or customizing workflow execution.

Create a snakemake-profile.yaml file with content like:

executor: local # clusters e.g slurm, lsf, aws-batch ... see snakemake documentation 
jobs: "30"
max-jobs-per-second: "10"
max-status-checks-per-second: "10"
local-cores: 44
latency-wait: "30"
printshellcmds: "True"
rerun-incomplete: "False"
keep-incomplete: "True"
conda-cleanup-envs: "False"
dryrun: true
resources:
    mem_mb: 16000
    threads: 8
  • executor: Cluster scheduler (e.g., SLURM).
  • jobs: Maximum number of parallel jobs.
  • resources: Default resource limits for jobs.
  • dryrun: Set to true to perform a dry-run (no jobs will be executed).

For more details on these and other Snakemake parameters, see the Snakemake documentation.

To use this profile, run:

edentity --profile snakemake-profile.yaml --config_file params_config.yaml

Snakemake parameters can also be provided directly via the command line, but they must be specified in their long form (e.g., --jobs instead of -j). Command-line parameters take precedence over those defined in the profile configuration file or the default parameters.

For example, you can use both a profile configuration file and override specific parameters via the command line:

edentity --profile snakemake-profile.yaml --config_file params_config.yaml \
--jobs 50 --latency-wait 60 --until merge 

In this example:

  • The --config_file option specifies the parameters specific to eDentity, such as input directories, primers, and quality control settings.
  • The --profile option specifies the Snakemake profile configuration file, which controls the behavior of Snakemake, such as job execution, resource limits, and cluster settings.
  • The --jobs, --latency-wait, and --until parameters override the corresponding values in the profile configuration file.
  • Command-line parameters always take priority over the profile or default settings.

For a full list of options params:

edentity --help

Pipeline Output Directory Structure

After successful execution, the pipeline generates a structured set of output directories and files within your specified work_dir. All file names are prefixed with your work_dir. The main components are:

Full pipeline run

work_dir/
├── Results/
│   ├── ESVs_fasta/                          # FASTA file of Exact Sequence Variants
│   └── report/
│       ├── {work_dir}_ESV_table.tsv         # ESV abundance table
│       ├── {work_dir}_summary_report.tsv    # Per-sample summary statistics
│       ├── {work_dir}_multiqc_reports/
│       │   └── {work_dir}_multiqc.html      # Interactive MultiQC report
│       └── fastpQC/                         # Per-sample FastP JSON reports
├── logs/                                    # Log files for each pipeline step
│   ├── fastpQC/
│   ├── merge/
│   ├── trimming/
│   ├── filter/
│   ├── dereplication/
│   ├── denoise/
│   ├── chimera/
│   ├── search_exact/
│   └── multiqc/
└── edentity_pipeline_settings/              # Configuration files used for this run
    ├── {work_dir}_snakemake_config.yml
    ├── {work_dir}_snakemake_profile/
    └── multiqc_config/

QC-only mode (--qc_only)

work_dir/
├── Results/
│   └── report/
│       ├── {work_dir}_summary_report.tsv    # Per-sample QC summary
│       └── {work_dir}_multiqc_reports/
│           └── {work_dir}_QC_only.html      # Interactive MultiQC QC report
├── logs/
│   ├── fastpQC/
│   ├── merge/
│   └── multiqc/
└── edentity_pipeline_settings/

QC-only + plate summary (--qc_only --plate_summary)

work_dir/
├── Results/
│   └── report/
│       ├── {work_dir}_summary_report.tsv
│       ├── {work_dir}_QC_only.html          # MultiQC QC report
│       └── {work_dir}_plate_yield.html      # Self-contained plate yield HTML report
├── logs/
│   ├── fastpQC/
│   ├── merge/
│   ├── multiqc/
│   └── plate_summary/
└── edentity_pipeline_settings/

Standalone plate yield (--plate_summary_only)

Reads {work_dir}_summary_report.tsv from an existing run and writes one new file:

work_dir/
└── Results/
    └── report/
        └── {work_dir}_plate_yield.html      # Self-contained plate yield HTML report

Metadata

Release files for edentity 1.6.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for edentity 1.6.1
File Size Uploaded Provenance Actions
edentity-1.6.1.tar.gz 93.0 kB Without provenance Details for edentity-1.6.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for edentity 1.6.1
File Interpreter ABI Platform Actions
edentity-1.6.1-py3-none-any.whl Python 3 none any Details for edentity-1.6.1-py3-none-any.whl

Total release size: 153.1 kB

Release files / edentity-1.6.1.tar.gz

Download URL edentity-1.6.1.tar.gz
Size 93.0 kB
Tags Source
SHA-256 checksum
How to use checksums
cc1db022d210813d4d9937610c713c5dfdf676c4311482af155732ba2df2a52e
BLAKE2b-256 checksum
How to use checksums
87f772c3bf186b52cf305d979fd7c0ed667dae42a68516ae78bea5258af7c7cf
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.22

Release files / edentity-1.6.1-py3-none-any.whl

Download URL edentity-1.6.1-py3-none-any.whl
Size 60.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
2e0fa7295ac4bfd73908f0fddc63622024502ec426a94c1aca5d456602c1bda6
BLAKE2b-256 checksum
How to use checksums
40f4425d21aabc31a385b2a667ad8579a1b2c1f8b04259be1c4a764257c4d4b3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.22

Security for edentity 1.6.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page