GANDALF

Graph Analysis Navigator for Discovery And Link Finding

A high-performance Python library and Translator-compatible TRAPI server for fast path finding in large biomedical knowledge graphs.

Features

• Compressed Sparse Row (CSR) graph representation for memory-efficient storage of 10M+ node, 38M+ edge graphs
• Bidirectional search for optimal path-finding performance
• O(1) property lookups via hash indexing
• Predicate filtering to reduce path explosion
• Qualifier filtering for advanced edge constraints (aspect, direction, mechanism)
• Attribute constraints on edges and nodes, including filtering edges by specific PubMed IDs
• Subclass expansion via Biolink Model Toolkit with configurable depth
• Batch property enrichment — enrich only final paths, not intermediate results
• Diagnostic tools to understand path counts and explosion
• TRAPI 2.0 compatible REST API with Plater-compatible endpoints, modelled with translator_tom
• Async query support with callback URLs
• Dehydrated mode for lightweight responses that skip edge and node attribute enrichment
• OpenTelemetry tracing with Jaeger integration

Installation

Recommended: Use a virtual environment

Some transitive dependencies (e.g., stringcase, pytest-logging) require modern pip/setuptools to build correctly. Using a virtual environment ensures you have updated tools.

# Create and activate a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Upgrade pip and setuptools (important for building dependencies)
pip install --upgrade pip setuptools wheel

# Install the core package
pip install -e .

# Install with server dependencies (FastAPI, uvicorn, etc.)
pip install -e ".[server]"

# Install with dev dependencies (pytest, black, mypy)
pip install -e ".[dev]"

Quick Start

Unzipping a full Translator KGX

tar -xvf translator_kg.tar.zst

This will output a nodes.jsonl and edges.jsonl file.

Build a graph from JSONL

from gandalf import build_graph_from_jsonl

# Build with ontology filtering
graph = build_graph_from_jsonl(
    edges_path="data/raw/edges.jsonl",
    nodes_path="data/raw/nodes.jsonl",
)

# Save for fast loading
graph.save_mmap("data/processed/gandalf_mmap")

Upgrading to TRAPI 2.0 requires a rebuild. Some of what 2.0 mandates is baked into the serialized graph, so a graph built by an earlier version cannot serve a conformant response. CSRGraph.load_mmap refuses such a graph outright with a GraphFormatError naming the rebuild, rather than starting up and serving edges with no knowledge_level / agent_type.

Baked in at build time (a rebuild is the only way to change these): knowledge_level / agent_type on every edge, Node.name omitted when unknown, Node.categories defaulted to biolink:NamedThing, RetrievalSource.upstream_resource_ids omitted when empty, and the persisted meta_kg.json / sri_testing_data.json.

Everything else 2.0 changed is computed per query and takes effect on deploy: binding shapes, QEdge.constraints, the parameters object and its timeout, the response envelope, and the remaining null / empty-container rules.

Query paths (TRAPI format)

from gandalf import CSRGraph, lookup

# Load graph (takes ~1-2 seconds)
graph = CSRGraph.load_mmap("data/processed/gandalf_mmap")

# Execute a TRAPI query
response = lookup(
    graph,
    {
        "message": {
            "query_graph": {
                "nodes": {
                    "n0": {"ids": ["CHEBI:45783"]},
                    "n1": {"categories": ["biolink:Gene"]},
                    "n2": {"categories": ["biolink:Disease"]}
                },
                "edges": {
                    "e0": {"subject": "n0", "object": "n1", "predicates": ["biolink:affects"]},
                    "e1": {"subject": "n1", "object": "n2"}
                }
            }
        }
    },
    subclass=True,
    subclass_depth=1,
)

print(f"Found {len(response['message']['results'])} paths")

Constraining query edges

TRAPI 2.0 gathers every constraint on a query edge into one constraints object. All constraints given must hold:

"edges": {
    "e0": {
        "subject": "n0",
        "object": "n1",
        "predicates": ["biolink:affects"],
        "constraints": {
            "knowledge_level": {
                "behavior": "ALLOW",
                "values": ["knowledge_assertion"],
            },
            "agent_type": {"behavior": "DENY", "values": ["text_mining_agent"]},
            "sources": {
                "behavior": "ALLOW",
                "values": ["infores:ctd"],
                "primary_only": True,
            },
            "qualifiers": [
                {"biolink:object_aspect_qualifier": "activity"},
            ],
            "attributes": [
                {"id": "biolink:publications", "operator": "==", "value": [...]},
            ],
        },
    }
}

• knowledge_level / agent_type — allow or deny Biolink values on the bound edges. ALLOW needs at least one listed value to be present; DENY needs none of them to be.
• sources — the same allow/deny over the infores CURIEs in an edge's sources. primary_only narrows the check to the source whose role is primary_knowledge_source, so a constraint can ignore aggregators.
• qualifiers — a list of qualifier mappings. AND within one mapping, OR between them. Values expand through the Biolink hierarchy, so a query for a parent value also matches edges carrying a child value.
• attributes — attribute constraints, evaluated against the edge's attributes. Query nodes accept the same list directly under constraints. "not": true negates one.

Migrating from TRAPI 1.x: qualifier_constraints is now constraints.qualifiers (and its qualifier_set list of qualifier_type_id/qualifier_value pairs collapses into a single mapping), and attribute_constraints is now constraints.attributes. Gandalf rejects the old field names with a 400 rather than ignoring them, so a stale client never silently receives unfiltered results.

Filtering edges by PubMed ID

Attribute values are often lists — publications above all — and every operator except === is applied to each member, so == reads as "contains". Filtering an edge down to specific PubMed IDs is therefore plain equality:

"constraints": {
    "attributes": [
        {
            "id": "biolink:publications",
            "operator": "==",
            "value": ["PMID:23456789", "PMID:11111111"],
        }
    ]
}

Only edges citing at least one of those PMIDs survive. A list value means "any of"; a single string constrains to one publication. Publication identifiers are compared canonically, so PMID:23456789, pubmed:23456789, https://pubmed.ncbi.nlm.nih.gov/23456789 and the bare 23456789 all select the same article — unlike the matches operator, which does a substring regex and would also accept PMID:234567890.

Architecture

The package uses a three-stage pipeline:

1. Topology Search (fast) - Find all paths using indices only
2. Filtering (medium) - Apply business logic on necessary node or edge properties
3. Enrichment (batch) - Load all properties for final paths only

This separation allows filtering millions of paths before expensive property lookups.

REST API

The server exposes Plater-compatible TRAPI endpoints on port 6429.

Run the development server:

python gandalf/main.py

Run the production server:

gunicorn gandalf.server:APP -c gunicorn.conf.py

Endpoints

Both /query and /asyncquery accept a single optional query parameter:

• ?profile=true — Emit per-stage timing diagnostics into message.logs

All other request configuration lives under the body's parameters object, which TRAPI 2.0 defines for query-time settings that do not change what a query means. The server repeats it back in the response, as the spec requires.

{
  "message": { "query_graph": { ... } },
  "parameters": {
    "timeout": 60,
    "log_level": "INFO",
    "bypass_cache": false,
    "subclass": true,
    "subclass_depth": 1,
    "dehydrated": false,
    "filter_config": { "max_node_degree": 50 },
    "annotator_config": {}
  }
}

Responses never serialize a null, and never serialize an empty container for a property whose schema forbids one (Edge.qualifiers, RetrievalSource.upstream_resource_ids, message.auxiliary_graphs, logs, …) — TRAPI 2.0 is OpenAPI 3.1 and dropped nullable, so an absent value is an absent property. message.results is deliberately still [] when a query matched nothing, which is what 2.0 asks for.

TRAPI's own parameters:

• timeout (number): Seconds the client is willing to wait. When the budget is spent the query stops and the response carries a Timeout status with the logs from the work done. A negative value disables the server's default timeout (GANDALF_QUERY_TIMEOUT). A value below GANDALF_MIN_QUERY_TIMEOUT is refused up front with HTTP 409, since the server knows it cannot answer that fast.
• log_level (string): Least critical level of logs to return — ERROR, WARNING, INFO or DEBUG. (This moved here from the request's top level in TRAPI 2.0.)
• bypass_cache (bool): Accepted and has no effect — gandalf answers from its own graph and holds no query cache.

Gandalf's own parameters:

• subclass (bool): Enable biolink subclass inference (default true)
• subclass_depth (int): Maximum subclass_of hops (default 1)
• dehydrated (bool): Return the smallest useful response — edges carry only subject, object, predicate, knowledge_level and agent_type, with no attributes and no sources (auto-enabled for very large result sets). TRAPI 2.0 requires sources, so a dehydrated response is deliberately not schema-valid: the mode trades conformance for size, and rehydrating one (see rehydrate) restores a conformant response
• rehydrate (bool): When true, the server skips the graph lookup and only enriches the knowledge_graph already supplied in message — used to re-enrich a previously dehydrated response
• filter_config (object): Plugin-defined node filter settings (each NodeFilter plugin reads its own key)
• annotator_config (object): Per-request opt-in response-annotator settings (each key activates one annotator plugin)

CLI Commands

# Build a CSR graph from JSONL node/edge files
gandalf-build --edges data/edges.jsonl --nodes data/nodes.jsonl --output data/graph_mmap/

# Query paths from the command line
gandalf-query --graph data/graph_mmap/ --start "CHEBI:45783" --end "MONDO:0004979"

# Diagnose path explosion between two nodes
gandalf-diagnose --graph data/graph_mmap/ --start "CHEBI:45783" --end "MONDO:0004979"

Configuration

The server is configured via environment variables (prefixed with GANDALF_):

Core

Search Tuning

TRAPI

Server Identity

Automat Heartbeat

Observability

Docker

# Build the image
docker build -t gandalf .

# Run with a graph volume
docker run -p 6429:6429 \
  -v /path/to/graph:/data/graph \
  -e GANDALF_GRAPH_PATH=/data/graph \
  gandalf

Verifying the Server

# Check graph metadata
curl http://localhost:6429/metadata

# Browse the API docs
open http://localhost:6429/docs

Releases

• Make a release in GitHub to run a GitHub Action that pushes a gandalf to ghcr
• Run this on the mmap folder: tar -czvf gandalf_mmap_<date>.tar.gz gandalf_mmap
• Upload the tar.gz file to a public file server
• Update any helm charts and deploy