ENA bins and MAGs uploader

This repository allows to:

• Generate xmls and manifests necessary for genome submission
• Link the genomes you want to submit with the samples/runs used to generate them
• Upload bins and MAGs in fasta format to ENA (European Nucleotide Archive) with webin-cli

How it works

When you submit genomes to the ENA, you need to register a sample for every genome containing all the relevant metadata describing the genome and the sample of origin. The genome_uploader acts as the main linker to preserve sample metadata as much as possible. For every genome to register, you need an INSDC run or assembly accession associated to the genome in order for the script to inherit its relevant metadata. On top of those metadata, the script adds metadata specified by the user that are specific to the genome, like taxonomy, statistics, or the tools used to generate it. The metadata that ENA requires are descibed in the checklist for MAGs and for bins, respectively.

Prepare Input TSV

The genome_uploader takes as input one tsv (tab-separated values) table in the following format:

With columns indicating:

• genome_name: genome id (unique string identifier)
• accessions: run(s) or assembly(ies) the genome was generated from (DRR/ERR/SRRxxxxxx for runs, DRZ/ERZ/SRZxxxxxx for assemblies). If the genome was generated by a co-assembly of multiple runs, separate them with a comma.
• assembly_software: assemblerName_vX.X
• binning_software: binnerName_vX.X
• binning_parameters: binning parameters
• stats_generation_software: software_vX.X
• completeness: float
• contamination: float
• rRNA_presence: True/False if all among 5S, 16S, and 23S genes, and at least 18 tRNA genes, have been detected in the genome
• NCBI_lineage: full NCBI lineage - format: x;y;z;.... The same organism can be described in two different ways: either in tax ids (integers) or strings. For example, the lineage for E. coli can be:
  • Bacteria;Pseudomonadati;Pseudomonadota;Gammaproteobacteria;Enterobacterales;Enterobacteriaceae;Escherichia
  • 2;1224;1236;91347;543;561;562
• metagenome: needs to be listed in the taxonomy tree here (you might need to press "Tax tree - Show" in the right most section of the page)
• co-assembly: True/False, whether the genome was generated from a co-assembly. N.B. the script only supports co-assemblies generated from the same project.
• genome_coverage : genome coverage against raw reads
• genome_path: path to genome to upload (already compressed)
• broad_environment: string (explanation following)
• local_environment: string (explanation following)
• environmental_medium: string (explanation following)
• chromosome_level (optional): True/False, whether the genome should be submitted as chromosome-level genome assembly (bins can't be submitted as chromosome-level genome assembly). If the column is missing or the cell is empty, it defaults to False. Genomes with chromosome_level set to True must have their contigs listed in a separate TSV passed with --chromosome-info - see Submission of chromosome-level genome assembly below.

Columns marked True/False (rRNA_presence, co-assembly and chromosome_level) are not case sensitive and accept True/Yes/Y for true and False/No/N for false (e.g. true, YES and n are all valid). An empty cell counts as False where the column is allowed to be empty (rRNA_presence for bins, chromosome_level). Any other value stops the script with an error naming the column and the genomes.

An example of input tsv table can be found here.

Environment related arguments

According to ENA checklist's guidelines, broad_environment describes the broad ecological context of a sample - desert, taiga, coral reef, ... local_environment is more local - lake, harbour, cliff, ... environmental_medium is either the material displaced by the sample, or the one in which the sample was embedded prior to the sampling event - air, soil, water, ... For host-associated metagenomic samples, the three variables can be defined similarly to the following example for the chicken gut metagenome: "chicken digestive system", "digestive tube", "caecum". More information can be found at ERC000050 for bins and ERC000047 for MAGs under field names "broad-scale environmental context", "local environmental context", "environmental medium"

Warnings

Mandatory vs Optional Fields

All fields above are mandatory for MAG submission (see ENA's MAGs checklist here). However, if you are registering bins, you cand decide whether to omit the following fields: completeness, contamination and rRNA_presence(see ENA's bins checklist here). These values are used together to determine MAG quality according to MIMAG criteria (described here).

If you already generated these for your bins, our recommendation is to include them for shareability and to describe your sample more accurately.

Existing accessions in the INSDC

Raw-read runs or assemblies from which genomes were generated should already be available on the INSDC (ENA by EBI, GenBank by NCBI, or DDBJ) for this script to work. Therefore, at least a DRR|ERR|SRR accession (for runs) or a ERZ|SRZ|DRZ accession (for assemblies) should be available.

If you are working with your own, private data on ENA, you will need to add the --private flag to access private metadata through ENA API. This implies that if you are working on public data, you can omit the flag. However, you will need to submit two different batches of data if you are handling both private and public data.

Submission of chromosome-level genome assembly

By default genome_uploader submits input MAGs and bins as contig-level assemblies. High-quality MAGs, complete replicons (including plasmids, organelles, complete viral genomes, etc) can be submitted as chromosome-level assembly. Refer to the official ENA documentation for more details.

Genomes to submit this way have to be listed in a separate, optional TSV providing additional metadata passed with --chromosome-info - one row per each sequence in each assembly. Check ENA docs before populating this file. A genome can have several rows (e.g. one chromosome and one plasmid), all sharing the same genome_name, and each such genome must have chromosome_level set to True in the main --genome_info table. The file has these columns:

Example - a genome with one chromosome and plasmid:

TPA generation and upload

If uploading TPA (Third PArty) genomes, you will need to contact ENA support before using the script. They will provide instructions on how to correctly register a TPA project where to submit your genomes. If both TPA and non-TPA genomes need to be uploaded, please divide them in two batches and use the --tpa flag only with TPA genomes.

Compress your fasta files

Files to be uploaded will need to be compressed with gzip (.gz) or bzip2 (.bz2).

Split your input tables

No more than 5000 genomes can be submitted at the same time. If you have more than 5000, split your table into smaller ones and launch the genome_uploader for each table.

Installation

Installation with conda (recommended)

Command will install all necessary dependencies into genomeuploader environment

conda install bioconda::genome-uploader

Installation with pip

Install genome_uploader with:

pip install genome_uploader

Additionally, you need to download the webin-cli.jar from the latest release.

Setting ENA Credentials

This tool requires your ENA Webin credentials to function. You can provide these by setting environment variables or using an environment file.

Using an environment file

Create a file named .env in your home directory (~/.env), your current working directory (./.env), or specify a custom file (default is .env).

Add the following lines with your credentials:

ENA_WEBIN=your_username_here
ENA_WEBIN_PASSWORD=your_password_here

Alternatively, set the environment variables directly in your shell

Important

If your ENA password contains special characters (for example $, !, &, =), you must wrap it in single quotes when exporting it as an environment variable. Otherwise, your shell may interpret parts of the password and silently change its value, leading to authentication errors.

export ENA_WEBIN=your_username_here
export ENA_WEBIN_PASSWORD=your_password_here

Run

Generate files for upload

Run genome_uploader with input TSV:

genome_upload \
  -u UPLOAD_STUDY \
  --genome_info METADATA_FILE \
  (--mags | --bins) \
  [--centre_name CENTRE_NAME] \
  [--chromosome-info CHROMOSOME_FILE] \
  [--out] [--force] [--live] [--tpa]

where

• -u UPLOAD_STUDY: study accession for genomes upload to ENA (in format ERPxxxxxx or PRJEBxxxxxx)
• --genome_info METADATA_FILE : genomes metadata file in tsv format
• --chromosome-info CHROMOSOME_FILE: optional tsv listing genomes to submit as chromosomes. See Submission of chromosome-level genome assembly submission below.
• -m, --mags, --b, --bins: select either of these for bin or MAG upload. If in doubt, check which definition fits best according to ENA
• --out: output folder (default: working directory)
• --force: forces reset of sample xmls generation. This is useful if you changed something in your tsv table, or if ENA metadata haven't been downloaded correctly (you can check this in ENA_backup.json).
• --live: registers genomes on ENA's live server. Omitting this option allows to validate samples beforehand (it will need the -test option in the upload command for the test submission to work)
• --centre_name CENTRE_NAME: name of the center generating and uploading genomes. It must be provided for broker accounts. If omitted, ENA uses the account's registered center name. Check ENA documentation for more details.
• --tpa: if uploading TPA (Third PArty) generated genomes
• --private: if data is private

Sample xmls won't be regenerated automatically if a previous xml already exists. If any metadata or value in the tsv table changes, --force will allow xml regeneration.

Produced files:

The script produces the following files and folders:

bin_upload/MAG_upload
├── manifests                       # also contains a *_chromosome_list.txt per genome submitted as a chromosome
│    └── ...
├── manifests_test                  # folder generated for validation in test mode
│    └── ...
├── ENA_backup.json                 # backup file to prevent re-download of metadata from ENA. Regeneration can be forced with --force
├── genome_samples.xml              # xml generated to register samples on ENA before the upload
├── registered_bins/MAGs.tsv        # list of genomes registered on ENA in live mode - needed for manifest generation
├── registered_bins/MAGs_test.tsv   # list of genomes registered on ENA in test mode - needed for manifest generation
└── submission.xml                  # xml used for genome registration on ENA

An example of output files and folder structure submitted in test mode can be found under the examples folder.

Upload genomes

Once manifest files are generated, it is necessary to use ENA's webin-cli resource to upload genomes.

More information about ENA's webin-cli can be found in the ENA docs.

We recommend using a pre-installed webin_cli_handler script.

Run live execution:

webin_cli_handler \
  --manifest <alias>.manifest \
  --context genome \
  --mode submit \
  [--test]

If you do not have ena-webin-cli installed add the --download-webin-cli flag. The tool will be automatically downloaded. It requires a recent JAVA version to be able to work following official repo.
If you want to use your local Java .jar (downloaded after pip installation) provide it with --webin-cli-jar.

Other options:

webin_cli_handler 

  -h, --help            show this help message and exit
  -m, --manifest MANIFEST
                        Path to a single manifest file or a directory containing manifest files
  -o, --output-accessions OUTPUT_ACCESSIONS
                        File to write assigned accessions to (TSV, default: ena_accessions.tsv)
  -c, --context {genome,transcriptome,sequence,polysample,reads,taxrefset}
                        Submission type: genome, transcriptome, sequence, polysample, reads, taxrefset
  --mode {submit,validate}
                        submit or validate
  --test                Specify to use test server instead of live
  --fasta-dir FASTA_DIR
                        Path to the FASTA files declared in the manifest file(s), required if manifest(s) reference file(s) that cannot be accessed from the execution directory
  --outdir OUTDIR       Base output directory for webin-cli files
  --download-webin-cli  Specify if you do not have ena-webin-cli installed
  --download-webin-cli-directory DOWNLOAD_WEBIN_CLI_DIRECTORY
                        Path to save webin-cli into
  --download-webin-cli-version DOWNLOAD_WEBIN_CLI_VERSION
                        Version of ena-webin-cli to download, default: latest
  --webin-cli-jar WEBIN_CLI_JAR
                        Path to pre-downloaded webin-cli.jar file to execute
  --retries RETRIES     Number of retry attempts (must be >= 1, default: 3)
  --retry-delay RETRY_DELAY
                        Initial retry delay in seconds (must be >= 0, default: 5)
  --java-heap-size-initial JAVA_HEAP_SIZE_INITIAL
                        Java initial heap size in GB (-Xms); only added when explicitly provided
  --java-heap-size-max JAVA_HEAP_SIZE_MAX
                        Java maximum heap size in GB (-Xmx); only added when explicitly provided
  --debug               Enable debug logging
  --resume              Resume previous webin_cli_handler run: skip aliases already present in accessions output file

Devs section

Testing submission in normal mode vs strict submission

ENA's test servers reset every day. This means that if you try to register the same set of samples more than once in a single day, the request will fail because the automatically generated aliases would result as duplicates on ENA's servers. To prevent this issue, when you register samples in test mode, the genome_uploader appends a timestamp to each generated alias. This ensures that you can repeat your tests multiple times without running into duplicate-alias errors.

However, when debugging or checking the script’s behavior in development mode, you might want the aliases to remain consistent across runs, so that repeated submissions refer to the same sample. To allow this, you can use the --test-suffix flag when running genome_upload.py, which lets you define a custom suffix instead of the automatic timestamp. This gives you more control over how sample aliases are generated during testing.