ontology-loader 0.5.0

ontology_loader

A suite of tools to configure and load an ontology from the OboFoundary into the data object for OntologyClass as specified by the NMDC schema.

Architecture: MongoDB access patterns

MongoDBLoader reaches MongoDB through two paths simultaneously — a deliberate hybrid, not an oversight.

linkml-store. Used for schema-aware setup and for any path where per-document work is acceptable:

• Client(handle=...) / attach_database(...) — declarative connection that integrates with NMDC's LinkML schema tooling.
• db.create_collection(name, recreate_if_exists=False) — idempotent collection setup.
• collection.index(...) — idempotent index declaration on id, name (class collection) and (subject, predicate, object) (relation collection).
• _handle_obsolete_terms — per-item processing of the small obsolete subset.

Raw pymongo. Used only for the bulk-upsert phase, exposed via the lazy MongoDBLoader._py_db property:

• py_collection.bulk_write([UpdateOne(...upsert=True), ...], ordered=False).

Why both?

linkml_store.api.stores.mongodb.mongodb_collection.upsert (as of the version pinned here) iterates per-item with find_one followed by update_one/insert_one. For small ontologies, that's fine. For larger ones, it's too slow.

The pymongo path is a bypass, not a permanent split. Upstream issue linkml/linkml-store#77 tracks adding bulk_write support to linkml-store.

Development Environment

Pre-requisites

• =Python 3.12

• Poetry
• Docker
• MongoDB
• NMDC materialized schema
• ENV variable for MONGO_PASSWORD (or pass it in via the cli/runner itself directly)

% docker pull mongo
% docker run -d --name mongodb-container -p 27018:27017 mongo

MongoDB Connection Settings

When connecting to MongoDB, you need to set the correct environment variables depending on where your code is running:

1. When running from your local machine (CLI or tests):

   export MONGO_HOST=localhost
   export MONGO_PORT=27018
   export ENABLE_DB_TESTS=true
   export MONGO_PASSWORD="your_valid_password"
   

2. When running inside Docker containers:

   export MONGO_HOST=mongo
   export MONGO_PORT=27017
   

The Docker container networking uses container names (like 'mongo') for internal communication, while your host machine must use 'localhost' with the mapped port (27018).

Basic mongosh commands

% docker ps
% docker exec -it [mongodb-container-id] bash
% mongosh mongodb://admin:root@mongo:27017/nmdc?authSource=admin
% show dbs
% use nmdc
% db.ontology_class_set.find().pretty()
% db.ontology_relation_set.find().pretty()
% db.ontology_class_set.find( { id: { $regex: /^PO/ } } ).pretty()
% db.ontology_class_set.find( { id: { $regex: /^UBERON/ } } ).pretty()
% db.ontology_class_set.find( { id: { $regex: /^ENVO/ } } ).pretty()

Command line

% poetry install
% poetry run ontology_loader --help
% poetry run ontology_loader --source-ontology envo
% poetry run ontology_loader --source-ontology envo --source-ontology po --source-ontology uberon

Five flags:

• --source-ontology <name> — required, repeatable. Lowercase prefix (envo, po, uberon, ncbitaxon, …). Multiple ontologies are processed sequentially in the given order.
• --report-directory <dir> — TSV report destination (only used in meticulous mode). Defaults to a fresh temp directory.
• --mode {meticulous|fast-initial} — default meticulous. See "Modes" below.
• --closure {combined|isa|partof|all|none} — default combined. Repeatable; values combine. all and none are exclusive.
• --exclude-descendants-of <CURIE> — optional, repeatable. Omits the proper rdfs:subClassOf descendants of each CURIE while keeping the CURIE itself. See "Excluding subtrees" below.

Modes

• meticulous (default): Preserves 0.2.x behavior — pure linkml-store, per-item upsert, force-refresh of the pystow cache on every run, TSV reports (ontology_updates.tsv, ontology_inserts.tsv, ontology_relation_inserts.tsv) written to --report-directory. Use this for incremental updates of an already-loaded ontology.
• fast-initial: Maximum-throughput first-time install. Raw pymongo insert_many(ordered=False), no upsert, no pre-read, no report tracking, no TSV writes. Reuses the pystow cache if present (downloads only when missing). Reruns against an already-populated collection skip documents that already exist (logged, not duplicated) rather than failing, so this is safe to retry. The one prerequisite: the target collection must already be free of duplicate id / (subject, predicate, object) values, since a genuinely dirty collection fails the one-time unique-index build this mode relies on. Expected ~3-5x faster than meticulous on large ontologies (e.g. NCBITaxon's 2.7M classes + 54.7M relations).

Closure shorthands

• --closure combined (default): emits entailed_isa_partof_closure (rdfs:subClassOf ∪ BFO:0000050).
• --closure isa: emits entailed_isa_closure (rdfs:subClassOf only).
• --closure partof: emits entailed_partof_closure (BFO:0000050 only).
• --closure all: shorthand for --closure combined --closure isa --closure partof. Exclusive.
• --closure none: emit no ancestry closure, only direct relationships. Exclusive.

Repeat the flag to combine specific closures: --closure isa --closure partof emits both entailed_isa_closure and entailed_partof_closure.

Running the tests

% make test

Running the linter

% make lint

Python API

pip install nmdc-ontology-loader

from ontology_loader.ontology_load_controller import OntologyLoaderController
import tempfile

# Default: pure linkml-store + TSV reports (preserves 0.2.x behavior)
OntologyLoaderController(
    source_ontology="envo",                          # str or list[str]
    report_directory=tempfile.gettempdir(),          # only used in 'meticulous' mode
    mode="meticulous",                               # or 'fast-initial'
    closure="combined",                              # str or list[str]
).run_ontology_loader()

Fast first-time install of a large ontology

OntologyLoaderController(
    source_ontology="ncbitaxon",
    mode="fast-initial",        # raw pymongo insert_many, no upsert, no reports
    closure="isa",              # is_a only; combined closure is too large for NCBITaxon
).run_ontology_loader()

Multiple ontologies in one invocation

OntologyLoaderController(
    source_ontology=["envo", "po", "uberon"],   # processed sequentially in given order
    mode="meticulous",
).run_ontology_loader()

Using with an existing MongoDB connection

If you already have a MongoDB connection (e.g., in a Dagster/Dagit job), pass it directly:

from pymongo import MongoClient
from ontology_loader.ontology_load_controller import OntologyLoaderController

mongo_client = MongoClient("mongodb://admin:password@localhost:27018/nmdc?authSource=admin")

OntologyLoaderController(
    source_ontology="envo",
    mode="meticulous",
    mongo_client=mongo_client,   # Pass the existing client
    db_name="nmdc",              # Required when passing an existing client
).run_ontology_loader()

Note: When passing an existing MongoDB client, you must also provide db_name. The database name cannot be auto-determined from a MongoClient instance.

Migrating from 0.2.x

The 0.2.x constructor signature (source_ontology, output_directory, generate_reports, mongo_client, db_name) continues to work as deprecated aliases. The exact call site in nmdc-runtime's Dagster load_ontology op runs unchanged under 0.3.0; two DeprecationWarning lines appear in the logs as a nudge.

See CHANGELOG.md for the full release note and a side-by-side migration code sample.

Testing CRUD operations in a live MongoDB

The test suite follows a single convention: tests that need MongoDB run automatically when MongoDB and credentials are available; they skip gracefully when not.

In practice:

• Mock-only tests (e.g. tests/test_mock_mongodb_loader.py) run unconditionally — no MongoDB or credentials needed.
• Tests that exercise a live MongoDB are gated by MONGO_PASSWORD (and a few additionally require ENABLE_DB_TESTS=true as an extra safety check against accidental writes against unintended databases). When the gating env vars are unset, those tests skip with a clear reason; when they are set, the tests connect to the MongoDB pointed at by the rest of the MONGO_* env vars.

Required env vars when running the live-DB tests:

export MONGO_HOST=localhost
export MONGO_PORT=27017            # or whatever your local Mongo listens on
export MONGO_USERNAME=admin
export MONGO_PASSWORD="your_valid_password"
export MONGO_DB=nmdc               # read by the loader (see src/ontology_loader/mongo_db_config.py)
export MONGO_DBNAME=nmdc           # read by tests/test_ontology_class_null_values.py — currently a separate name from MONGO_DB
export ENABLE_DB_TESTS=true        # required by tests/test_ontology_load_controller.py

Then:

make test

Same command runs without the env vars; the DB-gated tests just skip. Mock-only tests still run either way. This is intended both to prevent accidental writes against a live database when env vars aren't deliberately set, and to make sure MONGO_PASSWORD is never hardcoded in the codebase.

Safety rules for DB-writing tests

Any test that writes or modifies MongoDB documents must follow these rules:

1. Use a dedicated scratch database or collection name — never the production names (nmdc, ontology_class_set, ontology_relation_set). The scratch name should be specific enough that it can't collide with real data (e.g. ontology_loader_smoke_test).
2. Verify the target does not already exist before writing — if it does, the test must fail loudly with a clear message so the developer investigates rather than silently overwriting unrelated data.
3. Clean up unconditionally at the end — wrap the test in try / finally so the cleanup runs even when assertions fail.

The smoke test tests/test_cli_smoke.py::test_controller_end_to_end_against_live_mongo shows the pattern.

What each live-DB test does

Reset collections in dev

docker exec -it nmdc-runtime-test-mongo-1 bash

mongosh mongodb://admin:root@mongo:27017/nmdc?authSource=admin
db.ontology_class_set.find({}).pretty()
db.ontology_relation_set.find({}).pretty()
db.biosample_set.find({}).pretty()
db.ontology_class_set.drop()
db.ontology_relation_set.drop()
db.ontology_class_set.countDocuments()
db.ontology_relation_set.countDocuments()

Excluding subtrees

Use --exclude-descendants-of NCBITaxon:6656 to omit the proper rdfs:subClassOf descendants of Arthropoda while retaining Arthropoda itself and its relations to ancestors. Repeat the option for multiple CURIEs; every explicitly named term is retained, including nested selections. Exclusion is independent of --closure and applies in both loading modes to classes and to relations with either endpoint excluded. This filters the current load; it does not delete documents from an earlier load.

The semsql source is opened read-only by both oaklib and the loader. Closure relations are excluded in SQL; classes and oaklib direct relationships use an in-memory set of excluded identifiers. Exclusion roots and counts are logged, not persisted. Omitting the option preserves the full load.