EGA - RefgenDetector

RefgenDetector is a bioinformatics tool that infers the reference genome assembly used to create aligment files (BAM/CRAM/header) and VCFs.

Aligment Files

It identifies major genome releases and derived assemblies across humans and multiple other species by analyzing contig names and lengths from the header. Benchmarking against 94 synthetic datasets achieved a 100% accuracy rate, while large-scale testing on 918,404 real-world files demonstrated 97.13% correctness, failing only when files’ headers are incomplete.

Description

RefgenDetector is able to infer the following reference genomes:

Primates

👤 Homo sapiens

• hg16
• hg17
• hg18
• GRCh37
• GRCh38
• T2T

🐒 Pan troglodytes

• pantro3_0
• Pan_troglodytes-2.1

🐵 Macaca mulatta

• Mmul10
• rheMac8
• rheMac3

Rodents

🐭 Mus musculus

• mm7
• mm8
• mm9
• mm10
• mm39

🐀 Rattus norvegicus

• mRatBN7_2
• Rnor_6_0

Other Mammals

🐷 Sus scrofa

• Sscrofa10_2
• Sscrofa11_1

Vertebrates (Non-Mammalian)

🐟 Danio Rerio

• danRer10
• danRer11

Invertebrates

🪰 Drosophila Melanogaster

• dm5
• dm6

🐛 Caenorhabditis elegans

• WBcel215
• WBcel235

Microorganisms & Plants

🧫 Escherichia coli

• ASM886v2
• ASM584v2

🌱 Arabidopsis thaliana

• TAIR

🍺 Saccharomyces cerevisiae

• R64

ref_manager.py - Customize the assemblies database.

ref_manager.py provides command-line management of reference genomes used by RefgenDetector. It allows users to add custom assemblies from FASTA index (.fai) files, list all available references, and remove previously added custom entries without modifying the source code.

Usage

python ref_manager.py <command> [options]

Commands

Add a reference

python ref_manager.py add <genome.fai> <reference_name> <species> # script
refgenDetector-manager add <genome.fai> <reference_name> <species> # pip installation

Registers a new reference from a valid .fai file. If the contig structure matches an existing reference, the entry is not added.

List references

python ref_manager.py list # scripts
refgenDetector-manager list # pip installation

Displays all available references, including both built-in and user-defined assemblies.

Remove a reference

python ref_manager.py remove <reference_name> # scripts
refgenDetector-manager remove <reference_name> # pip installation

Removes a custom reference from the local database. Built-in references cannot be removed.

Notes

• Custom references are stored separately from the default reference database.
• Input files must be valid FASTA index files generated with samtools faidx.
• Duplicate assemblies are detected based on exact contig composition.

Variant Calling Files (VCFs)

From VCF files only 4 human assemblies can be inferred:

• Hg18
• GRCh37
• GRCh38
• T2T

Two different sources of information are used to infer the reference genome from variant calling files

• Header

In the VCF specification it is recommended, but not mandatory that the VCF header includes tags describing the reference and contigs backing the data contained in the file. When present, the tool will analyze this information and output the reference genome version based on the contig lengths, following the same logic of the aligment files inference.

• Variants

To infer the reference genome from a VCF the tool will read the VCF file in chunks of 100.000 variants, avoiding to load the complete file in memory. The POS and REF columns will be extracted and compared to the msgpack files.

The msgpack files were created comparing the nucleotides in each position for hg18, GRCh37, GRCh38 and T2T. Each file contains a list of the positions where each reference had a different nucleotide (distinguishing positions).

By getting the number of matches between these distinguishing positions and the REF present in the VCF we infer the reference genome version used to call the variants.

Bear in mind that the input VCF must be sorted to improve runtime performance.

Requirements

• Python 3.10.6

Depending on how you want to install the package:

• pip
• Docker

Installation

Cloning this repository

1. Clone this repository

cd refgenDetector
pip install -e . 

2. $ python3 refgenDetector_main.py -h

3. Download the msgpack files for the inference with VCFs: Download the msgpack reference

4. Move the msgpack to the correct path:

mv msgpacks.zip /refgenDetector/src/refgenDetector/
unzip /refgenDetector/src/refgenDetector/msgpacks.zip

From pypi

$ pip install refgenDetector

The reference files for the VCF inference will be automatically downloaded after the first execution of refgenDetector

With Docker

refgenDetector can be run using Docker, allowing it to work on Linux, Windows, and macOS without installing Python or other dependencies locally.

1. Requirements

• Docker with Docker Compose installed
• The refgenDetector repository cloned locally

2. Build and start the container

From the root directory of the repository, run:

docker compose up -d --build

This builds the Docker image and starts the container in the background.

For subsequent starts, use:

docker compose up -d

3. Add input files

Place the files you want to analyse in the data/ directory of the repository.

This folder is automatically mounted inside the container, so files can be added or removed without rebuilding it.

4. Run refgenDetector

Execute refgenDetector using:

docker compose exec refgendetector refgenDetector -t VCF -f sample.vcf.gz

Replace sample.vcf.gz with your input filename.

Supported input types include:

• VCF — VCF files
• BAM/CRAM — BAM or CRAM files
• Header — extracted BAM/CRAM headers

For example, to analyse a BAM file:

docker compose exec refgendetector refgenDetector -t BAM/CRAM -f sample.bam

To display the available options:

docker compose exec refgendetector refgenDetector -h

Note: On the first execution requiring reference databases, refgenDetector automatically downloads them. These files are stored in a persistent Docker volume and reused in subsequent executions, including after rebuilding the container.

5. Stop the container

docker compose down

This stops and removes the container while preserving the downloaded reference databases.

To start it again:

docker compose up -d

Important: Avoid using docker compose down -v, as this will also remove the persistent volume containing the reference databases.

Usage

Bear in mind that the VCF must be sorted for optimal execution.

You can get the help menu by running:

$ refgenDetector -h

usage: INFERRING THE REFERENCE GENOME USED TO ALIGN BAM OR CRAM FILE [-h] -f FILE -t {BAM/CRAM,Header,VCF,BIM} [--md5] [-a] [-v MAX_N_VAR] [-m MATCHES] [-r]

optional arguments:
  -h, --help            show this help message and exit
  -f FILE, --file FILE  Input file path
  -t {BAM/CRAM,Header,VCF,BIM}, --type {BAM/CRAM,Header,VCF,BIM}
                        Type of files to analyze.
  --md5                 Print md5 values if present in header.
  -a, --assembly        Print assembly if present in header.
  -v MAX_N_VAR, --max_n_var MAX_N_VAR
                        Maximum number of variants to read before stopping inference. The file is processed in chunks of 100,000 variants, so this value must be a multiple of 100,000 (e.g. 100000,
                        200000, 300000, ...).
  -m MATCHES, --matches MATCHES
                        Number of matches required before stopping. [DEFAULT:5000]
  -r, --resources       When set, print execution time, CPU, memory, and disk I/O usage

Test RefgenDetector

In the folder examples you can find headers, alignment and variant files to test the working of RefgenDetector.

Licence and funding

RefgenDetector is released under GNU General Public License v3.0.

It was funded by ELIXIR, the research infrastructure for life-science data (ELIXIR Beacon Implementation Studies 2019-2021 and 2022-2023).