ENA Metagenome Assembly uploader

Upload of metagenome and metatranscriptome assemblies to the European Nucleotide Archive (ENA)

Pre-requisites:

• Metadata file. One per study. See tests/fixtures/test_metadata.csv for an example
• Compressed assembly fasta files in the locations defined in the metadata file

Set the following environmental variables with your webin details:

ENA_WEBIN

export ENA_WEBIN=Webin-0000

ENA_WEBIN_PASSWORD

export ENA_WEBIN_PASSWORD=password

Installation

Installation with conda (recommended)

conda install bioconda::assembly_uploader

Installation with pip

pip install assembly-uploader

Additionally, you need to download the webin-cli.jar from the latest release.

Usage

From the command line

Register study and generate pre-upload files

If you already have a registered study accession for your assembly files skip to step 3.

Step 1: generate XML files for a new assembly study submission

This step will generate a folder <STUDY>_upload and a project XML and submission XML within it:

study_xmls
  --study STUDY         raw reads study ID
  --library LIBRARY     metagenome, metatranscriptome, or mixed (if study contains metagenomic and metatranscriptomic data)
  --center CENTER       center for upload e.g. EMG
  --hold HOLD           hold date (private) if it should be different from the provided study in format dd-mm-yyyy. Will inherit the release date of the raw read study if not
                        provided.
  --tpa                 use this flag if the study is a third party assembly. Default False
  --publication PUBLICATION
                        pubmed ID for connected publication if available
  --private             use flag if your data is private

Step 2: submit the new assembly study to ENA

This step will submit the XML to ENA and generate a new assembly study accession identifier. Make sure to write down the newly generated study accession identifier!

submit_study
  --study STUDY         raw reads study ID
  --directory PATH      directory containing study XML
  --test                run test submission only

Step 3: make a manifest file for each assembly

This step will generate manifest files in the folder <STUDY>_upload for runs specified in the metadata file:

Metadata table

assembly_manifest
  --study STUDY         raw reads study ID
  --data DATA           metadata table
  --data-delimiter      DATA delimiter, default: comma
  --assembly_study ASSEMBLY_STUDY
                        pre-existing study ID to submit to if available. Must exist in the webin account
  --force               overwrite all existing manifests
  --private             use flag if your data is private
  --tpa                 use this flag if the study is a third party assembly. Default False

Step 4: upload assemblies

Once manifest files are generated, it is necessary to use ENA's webin-cli resource to upload the metagenome/metatranscriptome assemblies. More information on ENA's webin-cli can be found in the ENA docs.

We recommend using a pre-installed webin_cli_handler script.

Run live execution:

webin_cli_handler \
  --manifest *.manifest \
  --context genome/metatranscriptome \
  --mode submit \
  [--test]

If you do not have ena-webin-cli installed add the --download-webin-cli flag. The tool will be automatically downloaded. It requires a recent JAVA version to be able to work following official repo.
If you want to use local Java .jar provide it with --webin-cli-jar.

Other options:

webin_cli_handler

  -h, --help            show this help message and exit
  -m, --manifest MANIFEST
                        Path to a single manifest file or a directory containing manifest files
  -o, --output-accessions OUTPUT_ACCESSIONS
                        File to write assigned accessions to (TSV, default: ena_accessions.tsv)
  -c, --context {genome,transcriptome,sequence,polysample,reads,taxrefset}
                        Submission type: genome, transcriptome, sequence, polysample, reads, taxrefset
  --mode {submit,validate}
                        submit or validate
  --test                Specify to use test server instead of live
  --workdir WORKDIR     Path to working directory
  --download-webin-cli  Specify if you do not have ena-webin-cli installed
  --download-webin-cli-directory DOWNLOAD_WEBIN_CLI_DIRECTORY
                        Path to save webin-cli into
  --download-webin-cli-version DOWNLOAD_WEBIN_CLI_VERSION
                        Version of ena-webin-cli to download, default: latest
  --webin-cli-jar WEBIN_CLI_JAR
                        Path to pre-downloaded webin-cli.jar file to execute
  --retries RETRIES     Number of retry attempts (must be >= 1, default: 3)
  --retry-delay RETRY_DELAY
                        Initial retry delay in seconds (must be >= 0, default: 5)
  --java-heap-size-initial JAVA_HEAP_SIZE_INITIAL
                        Java initial heap size in GB (-Xms); only added when explicitly provided
  --java-heap-size-max JAVA_HEAP_SIZE_MAX
                        Java maximum heap size in GB (-Xmx); only added when explicitly provided

Optional step 5: publicly releasing a private study

release_study
  --study STUDY         study ID (e.g. of the assembly study)
  --test                run test submission only

From a Python script

This assembly_uploader can also be used a Python library, so that you can integrate the steps into another Python workflow or tool.

from pathlib import Path

from assembly_uploader.study_xmls import StudyXMLGenerator, METAGENOME
from assembly_uploader.submit_study import submit_study
from assembly_uploader.assembly_manifest import AssemblyManifestGenerator

# Generate new assembly study XML files
StudyXMLGenerator(
    study="SRP272267",
    center_name="EMG",
    library=METAGENOME,
    tpa=True,
    output_dir=Path("my-study"),
).write()

# Submit new assembly study to ENA
new_study_accession = submit_study("SRP272267", is_test=True, directory=Path("my-study"))
print(f"My assembly study has the accession {new_study_accession}")

# Create manifest files for the assemblies to be uploaded
# This assumes you have a CSV file detailing the assemblies with their assembler and coverage metadata
# see tests/fixtures/test_metadata.csv for an example
AssemblyManifestGenerator(
    study="SRP272267",
    assembly_study=new_study_accession,
    assemblies_csv=Path("/path/to/my/assemblies.csv"),
    output_dir=Path("my-study"),
).write()

The ENA submission requires webin-cli, so follow Step 4 above. (You could still call this from Python, e.g. with subprocess.Popen.)

Finally, you can also publicly release a private/embargoed/held study:

from assembly_uploader.release_study import release_study
release_study("SRP272267")

Development setup

Prerequisites: a functioning conda or pixi installation.

To install the assembly uploader codebase in "editable" mode:

conda env create -f requirements.yml
conda activate assemblyuploader
pip install -e '.[dev,test]'
pre-commit install

Testing

pytest