GANDALF

Graph Analysis Navigator for Discovery And Link Finding

A high-performance Python library and Translator-compatible TRAPI server for fast path finding in large biomedical knowledge graphs.

Features

• Compressed Sparse Row (CSR) graph representation for memory-efficient storage of 10M+ node, 38M+ edge graphs
• Bidirectional search for optimal path-finding performance
• O(1) property lookups via hash indexing
• Predicate filtering to reduce path explosion
• Qualifier filtering for advanced edge constraints (aspect, direction, mechanism)
• Attribute constraints on edges and nodes, including filtering edges by specific PubMed IDs
• Subclass expansion via Biolink Model Toolkit with configurable depth
• Batch property enrichment — enrich only final paths, not intermediate results
• Diagnostic tools to understand path counts and explosion
• TRAPI 1.5 compatible REST API with Plater-compatible endpoints
• Async query support with callback URLs
• Dehydrated mode for lightweight responses that skip edge and node attribute enrichment
• OpenTelemetry tracing with Jaeger integration

Installation

Recommended: Use a virtual environment

Some transitive dependencies (e.g., stringcase, pytest-logging) require modern pip/setuptools to build correctly. Using a virtual environment ensures you have updated tools.

# Create and activate a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Upgrade pip and setuptools (important for building dependencies)
pip install --upgrade pip setuptools wheel

# Install the core package
pip install -e .

# Install with server dependencies (FastAPI, uvicorn, etc.)
pip install -e ".[server]"

# Install with dev dependencies (pytest, black, mypy)
pip install -e ".[dev]"

Quick Start

Unzipping a full Translator KGX

tar -xvf translator_kg.tar.zst

This will output a nodes.jsonl and edges.jsonl file.

Build a graph from JSONL

from gandalf import build_graph_from_jsonl

# Build with ontology filtering
graph = build_graph_from_jsonl(
    edges_path="data/raw/edges.jsonl",
    nodes_path="data/raw/nodes.jsonl",
)

# Save for fast loading
graph.save_mmap("data/processed/gandalf_mmap")

Query paths (TRAPI format)

from gandalf import CSRGraph, lookup

# Load graph (takes ~1-2 seconds)
graph = CSRGraph.load_mmap("data/processed/gandalf_mmap")

# Execute a TRAPI query
response = lookup(
    graph,
    {
        "message": {
            "query_graph": {
                "nodes": {
                    "n0": {"ids": ["CHEBI:45783"]},
                    "n1": {"categories": ["biolink:Gene"]},
                    "n2": {"categories": ["biolink:Disease"]}
                },
                "edges": {
                    "e0": {"subject": "n0", "object": "n1", "predicates": ["biolink:affects"]},
                    "e1": {"subject": "n1", "object": "n2"}
                }
            }
        }
    },
    subclass=True,
    subclass_depth=1,
)

print(f"Found {len(response['message']['results'])} paths")

Filtering edges by attribute (including PubMed IDs)

Any query edge accepts TRAPI attribute_constraints, evaluated against the edge's attributes; query nodes accept the same shape under constraints. Multiple constraints are ANDed, and "not": true negates one.

Attribute values are often lists — publications above all — and every operator except === is applied to each member, so == reads as "contains". Filtering an edge down to specific PubMed IDs is therefore plain equality:

"edges": {
    "e0": {
        "subject": "n0",
        "object": "n1",
        "predicates": ["biolink:affects"],
        "attribute_constraints": [
            {
                "id": "biolink:publications",
                "name": "publications",
                "operator": "==",
                "value": ["PMID:23456789", "PMID:11111111"],
            }
        ],
    }
}

Only edges citing at least one of those PMIDs survive. A list value means "any of"; a single string constrains to one publication. Publication identifiers are compared canonically, so PMID:23456789, pubmed:23456789, https://pubmed.ncbi.nlm.nih.gov/23456789 and the bare 23456789 all select the same article — unlike the matches operator, which does a substring regex and would also accept PMID:234567890.

Architecture

The package uses a three-stage pipeline:

1. Topology Search (fast) - Find all paths using indices only
2. Filtering (medium) - Apply business logic on necessary node or edge properties
3. Enrichment (batch) - Load all properties for final paths only

This separation allows filtering millions of paths before expensive property lookups.

REST API

The server exposes Plater-compatible TRAPI endpoints on port 6429.

Run the development server:

python gandalf/main.py

Run the production server:

gunicorn gandalf.server:APP -c gunicorn.conf.py

Endpoints

Both /query and /asyncquery accept a single optional query parameter:

• ?profile=true — Emit per-stage timing diagnostics into message.logs

All other request configuration lives under the body's parameters object:

{
  "message": { "query_graph": { ... } },
  "log_level": "INFO",
  "parameters": {
    "subclass": true,
    "subclass_depth": 1,
    "dehydrated": false,
    "filter_config": { "max_node_degree": 50 },
    "annotator_config": {}
  }
}

• subclass (bool): Enable biolink subclass inference (default true)
• subclass_depth (int): Maximum subclass_of hops (default 1)
• dehydrated (bool): Skip edge attribute enrichment for faster, lighter responses (auto-enabled for very large result sets)
• rehydrate (bool): When true, the server skips the graph lookup and only enriches the knowledge_graph already supplied in message — used to re-enrich a previously dehydrated response
• filter_config (object): Plugin-defined node filter settings (each NodeFilter plugin reads its own key)
• annotator_config (object): Per-request opt-in response-annotator settings (each key activates one annotator plugin)

CLI Commands

# Build a CSR graph from JSONL node/edge files
gandalf-build --edges data/edges.jsonl --nodes data/nodes.jsonl --output data/graph_mmap/

# Query paths from the command line
gandalf-query --graph data/graph_mmap/ --start "CHEBI:45783" --end "MONDO:0004979"

# Diagnose path explosion between two nodes
gandalf-diagnose --graph data/graph_mmap/ --start "CHEBI:45783" --end "MONDO:0004979"

Configuration

The server is configured via environment variables (prefixed with GANDALF_):

Core

Search Tuning

Server Identity

Automat Heartbeat

Observability

Docker

# Build the image
docker build -t gandalf .

# Run with a graph volume
docker run -p 6429:6429 \
  -v /path/to/graph:/data/graph \
  -e GANDALF_GRAPH_PATH=/data/graph \
  gandalf

Verifying the Server

# Check graph metadata
curl http://localhost:6429/metadata

# Browse the API docs
open http://localhost:6429/docs

Releases

• Make a release in GitHub to run a GitHub Action that pushes a gandalf to ghcr
• Run this on the mmap folder: tar -czvf gandalf_mmap_<date>.tar.gz gandalf_mmap
• Upload the tar.gz file to a public file server
• Update any helm charts and deploy