taxutils

Utilities for working with NCBI taxonomic data, accession-to-taxon mappings, taxonomy branches, corrected ranks, and pathogen target taxa. A version is available in Rust with optimized command-line functions, taxutils-rs, also available for install with bioconda.

Install

Conda Formula

conda install bioconda::taxutils

Pip Formula

pip install taxutils

taxutils requires its native Rust extension. Published wheels bundle the extension; there is no slower Python fallback for accession database or FASTA operations. An unsupported platform therefore fails at installation/import instead of silently changing performance or behavior.

from taxutils import backend_info

print(backend_info())
# {'selected': 'rust', 'rust_version': '1.1.2', 'api_version': 8}

Building from a repository checkout or source distribution requires Rust. The extension resolves taxutils 1.1.2 or newer compatible releases directly from crates.io; a sibling taxutils-rs checkout is not required.

Setup

taxutils stores downloaded taxonomy files (names.dmp, nodes.dmp), pathogen target metadata, and accession-to-taxon mappings in a global save directory. Set TAXUTILS_GLOBALS before importing the package if you want to control where these files live:

export TAXUTILS_GLOBALS=/path/to/taxutils/saves

If TAXUTILS_GLOBALS is not set, taxutils defaults to ./taxutils/ in the current working directory. It is the only environment variable taxutils reads, and it is only a default: save_folder= overrides it per call.

tu = taxutils(save_folder="/path/to/taxutils/saves")

SQLite scratch files (including refresh staging tables and index-sort spills) are confined to a private .taxutils-sqlite-* directory inside the resolved save folder. This uses TAXUTILS_GLOBALS by default, with the usual explicit save_folder override; no additional environment variable or option is needed. There is no fallback to system temporary directories. If scratch cannot be created or written, the operation reports an error. Scratch directories are removed when connections close, including after errors and cancellation. A hard process kill can leave an operation's directory behind; these are not automatically swept because another process may still be using them.

Threads

Every parallel stage — gzip decoding, the accession database build and refresh, and the FASTA commands — takes its worker count from a single threads argument. threads=None (the default) uses all logical CPUs.

tu = taxutils(low_memory=False, threads=8)   # cap the database build at 8 workers

The FASTA commands take the same value as --threads:

taxutils filter -i in.fasta -o out.fasta --remove-taxids 9606 --threads 8

Each call builds its own thread pool, so the setting applies to that operation only and never to the host process. RAYON_NUM_THREADS is not consulted.

The first run downloads NCBI taxonomy files. Accession lookups also use the NCBI accession-to-taxon mapping, which is large. By default, taxutils uses low-memory mode and scans the compressed mapping directly. For faster repeated lookups, use low_memory=False to build or reuse a local SQLite database:

from taxutils import taxutils

tu = taxutils(low_memory=False)

Database construction is owned by taxutils-rs. Both NCBI dumps are already sorted by accession, so they are decompressed in parallel and merged into one ascending stream that fills the table in key order. The table is keyed on the accession itself (WITHOUT ROWID), so there is no second copy of every accession to store or sort, and the taxid index is covering for reverse lookups. The result is assembled beside the destination and installed atomically only once every row and index is complete.

Long backend calls are interruptible: Ctrl-C during a build or a lookup raises KeyboardInterrupt promptly, removes the partial database, and leaves any database already installed untouched.

Keeping the database current

Anything missing is downloaded and a missing or unusable database is built automatically, so a first run needs no flags. To pick up new NCBI data, pass refresh=True: it re-fetches the managed taxonomy files and applies only the accessions NCBI added, changed or withdrew, skipping the work entirely when the sources are unchanged.

tu = taxutils(low_memory=False, wgs=True, refresh=True)

This asks NCBI whether each source has changed, and if so applies only the difference. Both the incoming dumps and the stored table are ordered by accession, so a single lockstep pass classifies every row as an insert, an update, or a deletion; accessions withdrawn upstream are removed. When the sources are unchanged, or the server cannot be reached, nothing is downloaded and the database is left alone. Adding WGS to a GB-only database uses one ordered atomic rebuild instead of staging and applying hundreds of millions of row changes. An existing WGS gzip is reused unless refresh=True was requested.

The compressed NCBI mapping is always retained, because low-memory lookups scan it directly. If an installation will only ever use the SQLite database, the .accession2taxid.gz files can simply be deleted to reclaim the space; they are downloaded again only if a later low-memory lookup needs them.

Core usage

Core functions are listed here. See the example notebook for a fuller walkthrough.

# Build object
tu = taxutils(accessions=None, low_memory=True, targets_json=None, wgs=False, save_folder=None, threads=None, refresh=False, canonical=True)  # Build the taxonomy utility object.

# Accession parsing and mapping
tu.parse_accession(header_strings, version=True)          # Extract one accession per string.
tu.load_a2t(accessions, low_memory=None, extend=False, wgs=None)  # Load accession-to-taxon mappings into tu.a2t.
tu.get_t2a(taxa, low_memory=None, wgs=None)                       # Return accessions assigned to taxa.

# Tree queries
tu.get_branch(taxon)                                      # Return the root-to-taxon branch.
tu.get_subtree(taxon)                                     # Return taxon plus all descendants.
tu.get_ancestor(taxon, anchor_rank)                       # Return nearest ancestor at a rank.
tu.is_leaf(taxon_or_taxa)                                 # Test whether taxa have no child nodes.
tu.is_child(taxon_a, taxon_b)                             # Test whether taxon_a is a direct child of taxon_b.
tu.is_descendent(taxon_a, taxon_b)                        # Test whether taxon_a is below taxon_b.
tu.get_lca(taxon_a, taxon_b)                              # Return the lowest common ancestor.
tu.get_distance(taxon_a, taxon_b)                         # Return tree edge distance through the LCA.
tu.sort_taxa(taxa)                                        # Sort taxa in taxonomic order.
tu.format_tree(taxa)                                      # Return an indented tree Series.
tu.topology(taxon, anchor_rank=None)                      # Return subtree topology metrics.
tu.topology(taxon, anchor_rank=None, stat="topology_scale")  # Return one topology statistic.

# Rank utilities
tu.get_rank_order()                                       # Return canonical rank codes.
tu.higher_than_rank(taxa, rank)                           # Test whether taxa are higher than a rank.

In taxutils, accessions=list/of/accessions can be passed to call load_a2t on construction of the taxutils object. A custom targets_json can similarly be passed in lieu of the default json explained below. A missing or unusable database is built from scratch automatically; refresh=True re-fetches the managed taxonomy files and updates an existing database in place, applying only the rows that changed upstream. By default, accession lookups use nucl_gb.accession2taxid.gz; pass wgs=True to also download/use nucl_wgs.accession2taxid.gz for WGS/TSA accessions. SQLite mode always uses nucl.accession2taxid.db; if it was built GB-only, a later wgs=True call upgrades the same DB with WGS mappings. load_a2t overwrites tu.a2t by default; pass extend=True to add missing mappings without discarding existing ones. Method-level low_memory=None and wgs=None use the modes set when tu was built. save_folder and threads are recorded on tu and reused by every later lookup.

parse_accession accepts strings, lists, arrays, and pandas Series. It returns the first accession found from each string using the same container type where possible; missing accessions are returned as "NA".

get_lca(a, b) returns the lowest common ancestor of two taxa. get_distance(a, b) returns the edge distance between two taxa through their lowest common ancestor. Depths are cached lazily as these methods are called.

get_ancestor(taxon, anchor_rank) returns the nearest ancestor at the requested corrected rank. If the input taxon already has that rank, it returns the input taxon; if no ancestor has that rank, it returns the input taxon as a fallback. It accepts a single taxon, list-like input, NumPy arrays, or pandas Series and returns the same container type where possible.

is_leaf(taxon) returns whether a taxon has no child nodes in the taxonomy tree. It accepts a single taxon, list-like input, NumPy arrays, or pandas Series. A single taxon returns a bool; a list-like input returns a list of booleans; a NumPy array returns a boolean array with the original shape; and a pandas Series returns a boolean Series with the original index.

is_child(taxon_a, taxon_b) returns whether taxon_a is a direct child of taxon_b. It uses the parent lookup directly, so each pairwise check is O(1).

is_descendent(taxon_a, taxon_b) returns whether taxon_a is a strict descendant of taxon_b; a taxon is not considered a descendant of itself. The first call builds a cached tree interval index in O(n), and subsequent pairwise checks are O(1).

is_child and is_descendent accept either two scalar taxa or two list-like inputs of the same length. Scalar inputs return a bool; list-like inputs return a list of booleans; NumPy arrays return boolean arrays with the original taxon_a shape; and pandas Series return boolean Series with the original taxon_a index.

topology(taxon, anchor_rank=None, stat=None) returns subtree topology metrics such as taxon count, leaf fraction, depth, branchiness, and topology_scale. Pass anchor_rank="F" to summarize the nearest family-level ancestor. With stat=None, a single taxon returns a Series and a list, array, or Series returns a DataFrame.

Pass stat to return one topology metric. A single taxon returns a scalar; a list, array, or Series returns a Series indexed by taxon.

Topology columns

tu.topology(...) returns these columns:

• taxon: input taxon.
• name: input taxon name.
• rank_code: corrected rank code for the input taxon.
• anchor_taxon: subtree root used for the topology summary.
• anchor_name: anchor taxon name.
• anchor_rank_code: corrected rank code for the anchor.
• n_taxa: number of taxa in the anchor subtree.
• n_leaves: number of terminal taxa in the anchor subtree.
• max_depth: maximum number of edges below the anchor.
• mean_depth: average number of edges below the anchor.
• topology_scale: 95th percentile descendant depth, with minimum value 1.
• max_children: largest number of direct children from any node in the subtree.
• branching_taxa_fraction: fraction of subtree taxa with at least one child.
• top_child_fraction: fraction of the anchor subtree contained in its largest immediate child branch.

tu.target_taxa contains the default pathogen-derived target taxa. Use it directly for target filtering or movement checks.

Rust acceleration

The native backend comes from the taxutils Rust crate. It accelerates load_a2t, get_t2a, and the extract, clean, grep, and filter FASTA commands while preserving the Python APIs and return values. Runtime errors are not retried through Python, so a failed native file operation cannot be run twice accidentally. Long native FASTA operations release the GIL and respond to KeyboardInterrupt between bounded batches; atomic-output commands discard their temporary output when cancelled.

Editable installs also compile an optimized Rust extension. After updating the source, rerun python -m pip install -e . to rebuild it; an older debug extension can make large accession scans substantially slower.

pandas/NumPy-returning taxonomy methods remain implemented in Python. This avoids converting already-efficient in-memory containers merely to cross the Python/Rust boundary; additional batch methods will only move behind the native backend after container-specific benchmarks show a benefit.

Run python benchmarks/backend_benchmark.py for native FASTA throughput or python benchmarks/database_build_benchmark.py for native SQLite construction throughput.

Rank correction

taxutils keeps the raw NCBI rank in rank and adds corrected rank columns. Canonical ranks (R, D, K, P, C, O, F, G, S) are used as anchors only when they move deeper than the corrected parent rank. Noncanonical ranks such as no rank, clade, and other unusual labels inherit position from the tree. If a child would be ranked at the same or a higher level than its parent, it is assigned a subrank such as S2, S3, or F2. The canonical name for the corrected rank is stored in new_rank.

Target taxa

In ZarLab, we are working on metagenomics in the clinical setting, with the goal of creating an "agnostic diagnostic". We often want to look at broad array of taxa (tu.target_taxa) that could cause harm to people. In June 2024, CZI did the work of compiling a list of pathogenic taxa. I did the easy work of turning this into a json and uploading it to my website, so that it is available and easily accessed for all time (in case that link ever breaks). taxutils will extend the taxa list to include subtrees of each of those pathogenic taxa. It will additionally include SARS-CoV2, since it was excluded from CZI's list. If you find any other obvious, missing pathogens, please send me a note, so I can update my json. You can also update the target_taxa member variable yourself, or store an entirely different set of targets, if you wanted.

Contact

Author: Will O'Brien
Affiliation: Computer Science Department, UCLA
Email: wob@cs.ucla.edu

Alternative viral accession mappings

tu = taxutils(canonical=False, low_memory=False, threads=8)
tu.load_a2t(accessions)

canonical=True (the default) preserves NCBI accession assignments. With canonical=False, Rust matches influenza A strains first, then H/N genotypes, then falls back to the NCBI mapping. Strain matches remain eligible when genotype is missing. Constructor accession lookups, load_a2t, and get_t2a all use this policy; reverse lookups remove accessions from their former assignments. Strain and subtype parsing is internal and adds no columns to tu.nodes.

The first alternative-mode initialization downloads NCBI AllNuclMetadata and streams only accession, genotype, and strain into viral.metadata.csv.gz in the save folder. Canonical-only users do not need this download. Once present, refresh=True refreshes it alongside taxonomy resources. An unavailable or malformed alternative resource raises an error; an individual unmatched record uses its NCBI assignment.

Indexed mode stores only differing assignments (plus successful matches absent from NCBI accession mappings) in a2t_overrides, with a taxid index for reverse queries. It shares the existing database without rewriting canonical rows. Metadata, taxonomy, or canonical-source changes invalidate the derived table. No source labels or redundant fallback values are stored.

Low-memory mode creates no database. Forward lookup scans the trimmed gzip for requested accessions. Reverse lookup uses two metadata passes to find candidate accessions and resolve duplicate priorities without retaining every override. Rust caches private taxonomy matching dictionaries and processes bounded batches using the configured threads. Gzip scanning still has a sequential component; more workers are not guaranteed to improve small queries.

examples/benchmark_alternative_mappings.py measures synthetic preparation, lookup, storage, and process peak RSS without downloading NCBI data. Full upstream metadata and taxonomy performance depends on their size and match distribution.