markdup 0.0.12

Fast, accurate BAM deduplication with intelligent UMI detection and correct fragment detection

MarkDup

A fast, accurate BAM deduplication tool with intelligent UMI detection and correct fragment detection.

Why MarkDup?

Existing BAM deduplication tools suffer from buggy duplicate detection due to incorrect biological positioning and strand handling, poor performance especially with UMI-based deduplication, and inadequate UMI clustering that leads to over-merging. MarkDup solves these problems with correct fragment detection using proper strand-aware coordinate handling, significantly faster processing through optimized algorithms and parallel processing, smart UMI clustering that prevents over-merging with frequency-aware algorithms, and automatic detection that handles both UMI and non-UMI data without requiring different tools.

Key Features

• 🔬 UMI-based deduplication with intelligent extraction from query names or BAM tags
• 📍 Coordinate-based deduplication for files without UMIs
• 🧬 Correct fragment detection for strand-aware clustering (start-only, end-only, or full fragment)
• 🔄 Auto-detection of UMI presence and format
• 🧬 Strand awareness for forward/reverse strand reads
• 📏 CIGAR handling for reads with indels and complex alignments
• ⚖️ Frequency balancing to prevent over-clustering of high-frequency UMIs
• 🎯 Advanced clustering with edit distance and frequency-aware algorithms
• 🔧 Quality selection with multiple metrics and automatic fallback
• ⚡ Parallelized processing for multi-core performance
• 📊 Comprehensive statistics and progress tracking

Quick Start

# Install
pip install markdup

# Basic usage (auto-detects everything)
markdup input.bam output.bam

# With multiple threads
markdup input.bam output.bam --threads 8

# Force coordinate-based (no UMIs)
markdup input.bam output.bam --no-umi

Key Features

🧬 Correct Biological Positioning

• Strand-aware coordinates: Properly handles forward/reverse strand reads
• CIGAR-aware positioning: Correctly processes indels and complex alignments
• Biological start/end: Uses 5'/3' positions, not reference positions

⚡ High Performance

• 13-113x faster UMI clustering with optimized algorithms
• Parallel processing: Multi-core support for large files
• Memory efficient: Window-based processing for large datasets

🔄 Automatic Detection

• UMI auto-detection: Finds UMIs in read names or BAM tags
• Sequencing type: Automatically detects single-end vs paired-end
• Quality metrics: Selects best quality criteria automatically

🎯 Smart UMI Clustering

• Frequency-aware: Prevents over-clustering of high-frequency UMIs
• Edit distance: Configurable similarity thresholds
• Exact matching: Handles identical UMIs efficiently

Installation

# From PyPI
pip install markdup

# From source
git clone https://github.com/y9c/markdup.git
cd markdup
pip install .

Usage Examples

Basic Deduplication

# Auto-detect UMIs and process
markdup input.bam output.bam

# Force coordinate-based method
markdup input.bam output.bam --no-umi

Advanced Options

# Custom UMI settings
markdup input.bam output.bam --umi-tag UB --min-edit-dist-frac 0.15

# Start-only positioning (ChIP-seq)
markdup input.bam output.bam --start-only

# Keep duplicates and mark them
markdup input.bam output.bam --keep-duplicates

Performance Tuning

# Use 8 threads
markdup input.bam output.bam --threads 8

# Larger windows for better performance
markdup input.bam output.bam --window-size 200000

Command Line Options

Option	Description	Default
INPUT_BAM	Input BAM file	Required
OUTPUT_BAM	Output BAM file	Required
--threads	Number of threads	1
--no-umi	Force coordinate-based	Auto-detect
--umi-tag	UMI BAM tag (e.g., UB)	Auto-detect
--start-only	Use start position only	False
--end-only	Use end position only	False
--keep-duplicates	Keep and mark duplicates	False
--min-edit-dist-frac	UMI edit distance threshold	0.1
--min-frequency-ratio	UMI frequency threshold	0.1

Output Format

MarkDup adds BAM tags to track deduplication:

Tag	Description
cn	Cluster name (chr:start-end:strand:UMI)
cs	Cluster size (number of reads)

Performance

• 13-113x faster UMI clustering with optimized algorithms
• Multi-core processing for parallel performance
• Memory efficient window-based processing for large files
• Automatic optimization based on input data characteristics

How It Works

1. Auto-detect: UMI presence, sequencing type, quality metrics
2. Group fragments: By biological position and strand
3. Cluster UMIs: Using edit distance and frequency-aware algorithms
4. Select best: Highest quality read from each cluster
5. Output: Deduplicated reads with cluster information

Documentation

• Installation Guide - How to install MarkDup
• Usage Guide - How to use MarkDup
• Algorithm Details - How MarkDup fixes existing problems
• FAQ - Frequently asked questions
• Contributing - How to contribute

Contributing

We welcome contributions! Please see our Contributing Guide for details.

License

MIT License - see LICENSE for details.

---

MarkDup: Fast, accurate BAM deduplication with intelligent UMI detection and correct fragment detection.