Aviary

An easy to use for wrapper for a robust snakemake pipeline for metagenomic short-read, long-read, and hybrid assembly. Aviary also performs binning, annotation, and provides users with an easy way to combine and dereplicate many aviary results with rapidity. The pipeline currently includes a series of distinct, yet flexible, modules that can seamlessly communicate with each other. Each module can be run independently or as a single pipeline depending on provided input.

Please refer to the full docs here

Quick Installation

Your conda channels should be configured ideally in this order:

conda config --add channels defaults
conda config --add channels bioconda
conda config --add channels conda-forge

Your resulting .condarc file should look something like:

channels:
  - conda-forge
  - bioconda
  - defaults

Option 1: Install from Bioconda

Conda can handle the creation of the environment for you directly:

conda create -n aviary -c bioconda aviary

Or install into existing environment:

conda install -c bioconda aviary

Option 2: Install from pip

Create the environment using the admin/requirements.txt file then install from pip:

conda env create -n aviary -f admin/requirements.txt
conda activate aviary
pip install aviary-genome

Option 3: Install from source

This method can be somewhat complicated, as you can see below, but it allows for easier/faster development and debugging of aviary itself.

Clone

To install from source, we recommend using pixi. First clone the aviary repository from GitHub:

git clone https://github.com/rhysnewell/aviary.git
cd aviary

Create aviary/.pixi (possibly symlinked to faster disk)

In the source directory, create the aviary/.pixi directory to hold pixi environments. Without this, .pixi is symlinked to aviary/.pixi within the checkout, which does not exist. So pixi run .. trips over that dead link.

This can be done by simply running:

mkdir -p aviary/.pixi

Post-installation of aviary

Then run postinstall so aviary can be run as a script. The postinstall task creates symlinks in the parent directory to allow running aviary directly via pixi run aviary ... in subsequent uses.

pixi run postinstall

When installed from source this way, aviary is installed in an "editable" way (similar to pip install -e .), meaning that any changes made to aviary source are immediately available via the aviary command. This is useful for development and debugging.

If you see something like the following error, you might have missed the previous step?

Error:   × Failed to create .pixi/ directory at .../aviary/.pixi
  ╰─▶ File exists (os error 17)

Running aviary from source

Then aviary can be run using pixi run (or via pixi shell).

pixi run aviary --help

Databases when running from source

As well as the standard method for defining database paths (e.g. CHECKM2DB), databases can be symlinked from a db/ directory in the aviary repository. An activation hook then ensures that these are available when in the pixi environments. To do this, create a db/ directory in the aviary repository and symlink the required databases into it. For example, as of writing:

$ ls db -l
lrwxrwxrwx - woodcrob 23 Apr 07:56 2.1.3 -> /mnt/hpccs01/work/microbiome/db/eggnog-mapper/2.1.3
lrwxrwxrwx - woodcrob 23 Apr 07:55 2015_01_16_v2 -> /work/microbiome/db/checkm/2015_01_16_v2
lrwxrwxrwx - woodcrob 23 Apr 07:54 CheckM2_database -> /work/microbiome/db/CheckM2_database/uniref100.KO.1.dmnd
lrwxrwxrwx - woodcrob 23 Apr 07:57 2024-3-28-GTDB214.1+humanT2T -> /work/microbiome/db/metabuli/2024-3-28-GTDB214.1+humanT2T/
lrwxrwxrwx - woodcrob 23 Apr 07:56 release232 -> /work/microbiome/db/gtdb/gtdb_release232/auxillary_files/gtdbtk_package/full_package/release232
lrwxrwxrwx - woodcrob 23 Apr 07:55 S6.5.0.GTDB_r232.metapackage_20260319.smpkg.zb -> /work/microbiome/db/singlem/S6.5.0.GTDB_r232.metapackage_20260319.smpkg.zb

To check the expected database symlink names, see admin/set_env_vars.sh in the aviary repository. The advantage of this approach is that locations of the databases are not tracked in the repository, which is advantageous as they are specific to the computing cluster of the user.

Option 4: Apptainer

This image bundles all of Aviary's tool dependencies, so make sure you have plenty of free disk space before pulling: Apptainer downloads the compressed layers, extracts them to a temporary uncompressed rootfs, and only then builds the final .sif — transient disk usage during that process can run to several tens of GB before it's cleaned up.

# 1. Point Apptainer's temp/build dir at real disk, not the default (often small/tmpfs)
mkdir -p ~/apptainer-tmp
export TMPDIR=~/apptainer-tmp

# 2. Force HTTP/1.1 — avoids long-stream resets some networks/proxies do to HTTP/2
export GODEBUG=http2client=0

# 3. Pull the image (retry if it drops mid-transfer — Apptainer caches completed
#    layers by digest, so a retry only re-fetches what failed)
apptainer pull aviary_0.13.3.sif docker://ghcr.io/snh-star/aviary:v0.13.3

# 4. Create a REAL, disk-backed overlay directory (not --writable-tmpfs —
#    that one is RAM-backed, small, and will fail mid-run with "No space left
#    on device" even with plenty of free disk). Apptainer's .sif filesystem
#    is read-only, and pixi (Aviary's environment manager) needs somewhere
#    writable at runtime.
mkdir -p ~/apptainer-overlay

# 5. Run aviary, attaching your real databases.
#
# The image bundles defaults for SINGLEM_METAPACKAGE_PATH, CHECKM2DB, and
# CHECKM_DATA_PATH already. GTDBTK_DATA_PATH, EGGNOG_DATA_DIR, and
# METABULI_DB_PATH are NOT bundled (those databases are too large) and must
# always be supplied — Aviary reads them at startup even for steps you don't
# run, so omitting them entirely causes a crash rather than a graceful
# default. If you don't need one of these steps, pass a placeholder (e.g.
# /tmp) instead of a real mount and use the matching skip flag where one
# exists (e.g. --skip-taxonomy for GTDB-Tk).
apptainer run \
  --overlay ~/apptainer-overlay:rw \
  --cleanenv \
  -B "$PWD":"$PWD" \
  -B /path/to/host/gtdbtk_data:/db/gtdb:ro \
  -B /path/to/host/eggnog_data:/db/eggnog:ro \
  -B /path/to/host/metabuli_data:/db/metabuli:ro \
  --env GTDBTK_DATA_PATH=/db/gtdb \
  --env EGGNOG_DATA_DIR=/db/eggnog \
  --env METABULI_DB_PATH=/db/metabuli \
  aviary_0.13.3.sif \
  <aviary-subcommand-and-args>

Checking installation

Whatever option you choose, running aviary --help should return the following output:

                    ......:::::: AVIARY ::::::......

           A comprehensive metagenomics bioinformatics pipeline

Metagenome assembly, binning, and annotation:
        assemble  - Perform hybrid assembly using short and long reads, 
                    or assembly using only short reads
        recover   - Recover MAGs from provided assembly using a variety 
                    of binning algorithms 
        annotate  - Annotate MAGs using EggNOG and GTBD-tk
        complete  - Runs each stage of the pipeline: assemble, recover, 
                    annotate in that order.
        cluster   - Combines and dereplicates the MAGs from multiple Aviary runs
                    using Galah

Isolate assembly, binning, and annotation:
        isolate   - Perform isolate assembly **PARTIALLY COMPLETED**
        
Utility modules:
        configure - Set or overwrite the environment variables for future runs.

Databases

Aviary uses programs which require access to locally stored databases. These databases can be quite large, as such we recommend setting up one instance of Aviary and these databases per machine or machine cluster.

The required databases are as follows:

• GTDB
• EggNog
• CheckM2
• SingleM

Optional databases include:

• Metabuli: for the optional TaxVAMB binner

Installing databases

Aviary can handle the download and installation of these databases via use of the --download flag. Using --download will download and install the databases into the folders corresponding to their associated environment variables. Aviary will ask you to set these environment variables upon first running and if they are not already available. Otherwise, users can use the aviary configure subcommand to reset the environment variables:

aviary configure -o logs/ --eggnog-db-path /shared/db/eggnog/ --gtdb-path /shared/db/gtdb/ --checkm2-db-path /shared/db/checkm2db/ --singlem-metapackage-path /shared/db/singlem/ --download

This command will check if the databases exist at those given locations, if they don't then aviary will download and change the conda environment variables to match those paths.

N.B. Again, these databases are VERY large. Please talk to your sysadmin/bioinformatics specialist about setting a shared location to install these databases to prevent unnecessary storage use. Additionally, the --download flag can be used within any aviary module to check that databases are configured properly.

Environment variables

Upon first running Aviary, you will be prompted to input the location for several database folders if they haven't already been provided. If at any point the location of these folders change you can use the aviary configure module to update the environment variables used by aviary.

These environment variables can also be configured manually, just set the following variables in your .bashrc file:

export GTDBTK_DATA_PATH=/path/to/gtdb/gtdb_release232/db/ # https://gtdb.ecogenomic.org/downloads
export EGGNOG_DATA_DIR=/path/to/eggnog-mapper/2.1.8/ # https://github.com/eggnogdb/eggnog-mapper/wiki/eggNOG-mapper-v2.1.5-to-v2.1.8#setup
export SINGLEM_METAPACKAGE_PATH=/path/to/singlem_metapackage.smpkg/
export CHECKM2DB=/path/to/checkm2db/uniref100.KO.1.dmnd

Workflow

Citations

If you use aviary then please be aware that you are using a great number of other programs and aviary wrapping around them. You should cite all of these tools as well, or whichever tools you know that you are using. To make this easy for you we have provided the following list of citations for you to use in alphabetical order. This list will be updated as new modules are added to aviary.

A constantly updating list of citations can be found in the Citations document.

License

Code is GPL-3.0