Domainator

A flexible and modular software suite for domain-based gene neighborhood and protein search, extraction, and clustering.

Domainator can be run as command line tools or as a Python library.

Note that the GUI that was bundled with earlier versions of Domainator has been removed. The preferred GUI solution for Domainator is the Rust rewrite Domainatoredux https://github.com/nebiolabs/domainatoredux.

Quickstart

conda create -n domainator -c conda-forge -c bioconda domainator python=3.12
conda activate domainator
domainator --help

domainator --help lists the commands, and domainator <command> --help describes one; domainator doctor lists the optional programs and which ones you have. This installs the released version from bioconda. See Installation to install from this repository instead, with every optional program.

Contents

• Overview and key concepts
• Program descriptions
• Requirements
• Installation
  • Conda
  • Apptainer (Singularity)
• Updating
• Examples
• Driving Domainator from other programs
• Using Domainator as a python library
• Credits

Detailed documentation

• The documentation website: these pages and the Python API reference
• Index
  • File Formats
  • Basic Examples
    • As a Google Colab notebook
  • Agent-driven workflows (see also AGENTS.md)
  • Advanced examples
  • Developing Domainator
  • Structure search and annotation
  • ESM-2 3B 3Di and foldseek integration
  • Limitations and frequently asked questions
  • Command-line reference
  • The tool manifest, for programs that drive Domainator

Questions, comments, bug reports, and feature requests

Please use our discussions page or create an issue

Overview and key concepts

Annotation tools (domainate, structure_domainate) annotate and retain every input record, including records without hits. Search tools (domain_search, structure_search) select matching records or regions. Both roles perform similarity searches; the distinction is what they emit, not whether an alignment is performed. Sequence search can return neighborhoods or translated CDSs; structure search returns matching targets' full observed protein sequences and marks their best-hit regions. Search best-hit annotations (Domain_Search features) are present by default; --add_annotations also adds domain annotations (Domainator features). See the command roles and output shapes.

For these four tools, -i identifies the records/database being annotated or searched, and -r supplies the references being searched for. Backend-specific "query" and "target" terminology does not reverse this CLI convention.

The three extract_* tools, select_by_cds, select_by_contig, and domain_search share taxonomy criteria; unresolved taxonomy is excluded by default, including with negated expressions. See taxonomy migration and examples and agent workflows.

Domainator provides several dozen discrete, flexible programs that can be composed into a broad range of workflows via command line or python scripting. Domainator also treats HMM-profiles as first-class citizens, supporting subsetting of .hmm files and comparison of HMM-profiles, including the construction of profile similarity networks and trees.

Domainator uses the GenBank file format as a carrier of both sequence and annotation data. Independence from a fixed set of sequence sources and the co-location of sequences and all their annotation data in a single file increases data portability and decreases complexity for end-users. Domainator can add functional annotations to sequences by comparison to protein HMM-profiles, protein sequences, nucleotide sequences, nucleotide HMMs, or infernal covariance models. Protein references are searched against CDSs or protein records, while nucleic-acid references are searched against whole nucleotide contigs. For example, in a single call to domainate, a set of genome or metagenome contigs can be annotated with hits to Pfam HMM-profiles, RFAM profiles, and hits to REBASE Gold Standard protein sequences at the same time.

The individual programs that make up Domainator can be roughly classified into six categories corresponding to their typical roles in workflows. The first steps in most workflows typically involve passing sequence data through one or more editors.

Editors are programs whose output format is the same as their input format. Each individual editor performs a simple task, but they can be combined in arbitrarily long chains to accomplish complex transformations. Examples are domain_search, which outputs a subset of the input sequences, based on the presence of a hit to a reference sequence or profile; domainate, which outputs all the input sequence but adds domain annotations based on hits to reference sequences; deduplicate_genbank, which performs similarity clustering using CD-HIT, usearch, or DIAMOND and outputs only the cluster representatives input sequences; and select_by_cds, which extracts genome neighborhoods around domains of interest.

In Domainator programs, perhaps differently from other software, the file being edited is supplied via the -i argument and criteria for editing is supplied via other arguments, such as the -r argument for reference sequences or hmm profiles. For example searching for hits to a query in the UniProt database may be accomplished via domainator domain_search -i uniprot.fasta -r query.hmm -o hits.gb, similarly annotating a set of contigs with Pfam annotations may be accomplished with domainator domainate -i unannotated.gb -r pfam-A.hmm -o annotated.gb .

Summary report programs summarize data into graphs and statistics, for example, the number of sequences in a file and the count of each kind of domain. Record-wise report programs produce tab-separated files, for example where each row corresponds to a genome contig, a protein, or a domain, and values are data such as length, taxonomy ID, domain content, etc.

Record-wise reports are useful for exporting data to programs, such as Excel, which can’t read GenBank or hmm files, and they also find use as intermediary files between some programs in Domainator.

Comparison programs generate pairwise score or distance matrices between contigs or HMM-profiles. Compare_contigs uses the Jaccard and adjacency indexes to compare proteins or gene neighborhoods based on their domain content, whereas seq_dist uses local alignment scores, for example via phmmer , Diamond , hmmsearch , or the Viterbi profile-comparison algorithm. Structure_dist compares protein structures directly by structural alignment, scoring them by alignment score, TM-score, lDDT or sequence identity. Comparison programs output data matrices.

Plotting programs convert data into formats appropriate for graphical visualization, for example converting score matrices and tabular metadata into trees or similarity networks which can be viewed in Cytoscape or other external visualization tools, depending on the data type.

Finally, there are few programs that defy categorization, these programs perform functions such as downloading data from NCBI or UniProt, converting files between formats, or generating profile-profile alignments.

Overview of Domainator components

*not all programs are shown

Example genome mining workflow

Example HMMER profile comparison workflow

Program descriptions

Every program is a command of the domainator program: domainator <command> [options], for example domainator domainate -i contigs.gb -r pfam.hmm -o annotated.gb. Run domainator --help for the list of commands, and domainator <command> --help for the documentation of each one. Every command also takes -v/--verbose (debug output, and a traceback for errors) and -q/--quiet (errors only, no progress messages or warnings).

To complete command names and options with the Tab key, install the shell completion script once (and again after upgrading Domainator):

# bash
domainator completion bash > ~/.local/share/bash-completion/completions/domainator
# zsh: write it to a directory on your $fpath, named _domainator
domainator completion zsh > ~/.zfunc/_domainator

Programs that can accept streaming input read genbank formatted data from stdin with -i -. Programs that need to pass over the input multiple times can't support streaming input. Programs that can produce streaming output write to stdout when an output is given as -, for example -o - or --fasta_out -. Nothing is read from stdin or written to stdout unless asked for. You can use shell pipes to move data from a program that supports streaming output into a program that supports streaming input, for example domainator domainate -i in.gb -r pfam.hmm -o - | domainator select_by_cds -i - --domains CcdB -o out.gb (but note that it is more efficient to use domainator as a python library, see below). Many commands that can generate large outputs also support --max_output_gb to fail early when a projected output would exceed the configured size limit. The default limit is 25 GB, and setting --max_output_gb 0 disables the guardrail.

Requirements

Domainator is developed on Linux, and has been tested on Mac. It probably won't work on Windows, except through WSL. It needs Python 3.12 or newer.

A few commands also need an external program or an optional Python package. The conda environment below installs all of them except esmologs and TMbed; domainator doctor lists which ones your installation has.

Domainator ships an optional native (Rust) extension that greatly speeds up GenBank/FASTA parsing and offset scanning. Building it requires a Rust toolchain (≥ 1.83) — which the conda environment installs for you — and a C compiler/linker (gcc/clang). If either is unavailable the extension is skipped and Domainator falls back to a slower pure-Python parser; results are identical either way. See Native (Rust) acceleration below.

Installation

There are two ways to install Domainator:

• The released version, from bioconda: conda create -n domainator -c conda-forge -c bioconda domainator, as in the Quickstart. Install any optional programs you need into the same environment (see Requirements).
• From this repository, with the conda environment below, which also installs the optional programs except esmologs and TMbed. Use this for the newest code or to develop Domainator.

NOTE: For Installation with ESM-2 3B 3Di and foldseek integration, see here

Conda

Step 0: Install miniconda

If you don't already have a conda distribution installed, install one, such as miniconda.

Miniconda

or

Mambaforge

Step 1: Download and install the domainator package

Clone the Domainator github repository and enter it:

git clone https://github.com/nebiolabs/domainator.git
cd domainator

Then create the conda environment, which installs Domainator and the external programs it uses:

On Linux:

conda env create -f conda_env.yml #x86_64 systems such as most linux machines, and older Apple machines.

On Mac:

conda env create -f conda_env_arm64.yml # ARM systems such as recent Apple machines, and a few linux machines.

This will create a new conda environment called "domainator".

Step 2: Activate the environment before you install packages or run any script

conda activate domainator

Optional: TMbed, for find_features --tmbed

find_features --tmbed predicts transmembrane regions and signal peptides with TMbed. The conda environment does not install it: TMbed is not on PyPI, and it brings in PyTorch, which is several GB. To use it, install it into the activated environment (or wherever you installed Domainator with pip):

conda activate domainator
pip install "transformers<5" "tmbed @ git+https://github.com/BernhoferM/TMbed.git@8cee893523eb655bc9485c00c65336d27a236191"
domainator doctor   # should now list torch and tmbed as found

• Python 3.14 or older. PyTorch has no Python 3.15 builds yet, so on 3.15 pip reports No matching distribution found for torch. Check with python --version; to make the environment on 3.14 instead, run conda create -n domainator python=3.14 and then conda env update -n domainator -f conda_env.yml in place of Step 1's conda env create.
• transformers<5: TMbed uses tokenizer methods that transformers 5 removed.
• The TMbed commit is pinned because TMbed has no releases; this is the version Domainator is tested with.
• GPU or CPU. --device (default cuda:0) falls back to the CPU when PyTorch can't use the GPU. pip's default PyTorch on Linux is built for a recent CUDA version (CUDA 13 for recent releases), which needs a recent NVIDIA driver; with an older driver, PyTorch can't use the GPU and TMbed runs on the CPU, much more slowly. domainator doctor reports whether PyTorch can use your GPU. If it can't, look up the "CUDA Version" at the top of nvidia-smi's output and install a PyTorch build for that version or older first, for example pip install torch --index-url https://download.pytorch.org/whl/cu124 for CUDA 12.4 (the builds are listed at https://pytorch.org/get-started/locally/). For a smaller CPU-only install, use --index-url https://download.pytorch.org/whl/cpu.
• Model download. The first run downloads TMbed's ProtT5 encoder (about 2.25 GB), by default into the installed tmbed package. Use --tmbed_model_dir (or --model_dir, or the HF_HOME environment variable) to keep it somewhere else, such as a shared directory.

Native (Rust) acceleration

Domainator includes an optional native extension (domainator._gbfast) that makes GenBank/FASTA parsing and offset scanning several times faster. It is optional: Domainator runs correctly without it, just more slowly (it falls back to a pure-Python parser), so this step can be skipped if you can't build it.

The conda environment already installs the Rust toolchain. You also need a C compiler/linker on your PATH:

• Linux: gcc — e.g. sudo apt install build-essential, or, inside the conda env, conda install -c conda-forge c-compiler
• macOS: the Xcode command line tools — xcode-select --install

With the environment activated, build the extension and confirm it loaded:

conda activate domainator
pip install --no-deps --force-reinstall .
python -c "from domainator import _gbfast; print('native (Rust) acceleration: ENABLED')"

If the import raises ModuleNotFoundError, the extension was not built — check that cargo/rustc and a C compiler are on your PATH — and Domainator will transparently use the slower Python parser instead.

Developing Domainator? Use an editable build instead so source edits take effect without reinstalling. The editable + build-isolation path does not compile the extension, so disable isolation and provide the build backend explicitly:

pip install "setuptools-rust>=1.7"
pip install -e . --no-build-isolation

Apptainer (Singularity)

Navigate to the directory containing this README file. It should also contain a file called domainator.def. In that directory execute the command:

apptainer build domainator.sif domainator.def

Or (if you have singularity not apptainer)

singularity build domainator.sif domainator.def

Then to execute domainator commands, prefix any command with

apptainer exec domainator.sif [command here]
# for example:
apptainer exec domainator.sif domainator domainate -i test/data/pDONR201_multi_genemark.gb -r test/data/pdonr_hmms.hmm -o example_out.gb

Updating

In the repository folder, get the new version and recreate the environment:

git pull
conda env create --yes -f conda_env.yml

This will delete the old domainator conda environment and replace it with the updated version.

If you use the native (Rust) acceleration, rebuild it after updating (pip install --no-deps --force-reinstall . with the environment activated).

Examples

Run Domainator to annotate contigs with Pfam

Activate the domainator conda environment

conda activate domainator

To run domainator on a single file

# download and extract Pfam
wget https://ftp.ebi.ac.uk/pub/databases/Pfam/current_release/Pfam-A.hmm.gz
gunzip Pfam-A.hmm.gz

# run Domainator
domainator domainate --cpu 4 --max_overlap 0.6 -i your_genbank.gb -r Pfam-A.hmm -o domainator_output.gb

Visualizing output

domainator summary_report -i domainator_output.gb --html_out domain_summary.html
domainator enum_report -i domainator_output.gb --by contig --columns taxname:domain:genus:species:self length domains --html_out enum_report.html -o enum_report.tsv
domainator plot_contigs -i domainator_output.gb --html_out contigs_plot.html

For more examples see: Basic Examples, and Advanced examples.

Driving Domainator from other programs

domainator describe prints a JSON description of every command and its options, which a GUI or a workflow system can render as forms and run with domainator <command> --config options.json. The browser GUI that used to be bundled (domainator_server) is becoming a separate project built on it. See docs/tool_manifest.md.

Using Domainator as a python library

Basics

Most executables in Domainator are based on a core function that acts as a editors on BioPython SeqRecord objects. So you can chain them within python scripts, keeping sequences as SeqRecords instead of writing and reading Genbank files. Chaining the functions will be similar to piping the output of one program to another, but will be faster because superfluous conversions from Genbank to SeqRecord and back again will be avoided. A caveat is that you lose some input and output validation (See also Interoperability with BioPython).

The general way to access these functions is from domainator.[script_name] import [script_name]. See the API reference for each function's parameters.

Interoperability with BioPython

Domainator's records are ordinary Biopython SeqRecord objects. For reading and writing Domainator-compatible sequence files, use domainator.parse_seqfiles and domainator.write_genbank: they keep qualifier values intact (Biopython's own GenBank writer wraps long values, which inserts spaces into them) and add Domainator's per-CDS annotation qualifiers.

Credits

Please cite

Johnson, Sean R., Peter Weigele, Alexey Fomenkov, Andrew Ge, Anna Vincze, James B. Eaglesham, Richard J. Roberts, and Zhiyi Sun. “Domainator, a Flexible Software Suite for Domain-Based Annotation and Neighborhood Analysis, Identifies Proteins Involved in Antiviral Systems.” 2024, Nucleic Acids Research, gkae1175. https://doi.org/10.1093/nar/gkae1175.

Projects that domainator draws inspiration from

• BigScape/Corason
• EFI
• pydna
• antiSMASH
• HMMER3
  • pyHMMER

Contributors

• Sean Johnson
• Peter Weigele
• Zhiyi Sun
• Andrew Ge
• Yu-Cheng Lin