omop-emb

Vector embedding layer for OMOP CDM concepts.

omop-emb generates, stores, and retrieves embeddings for OMOP concepts. It works out of the box with sqlite-vec (no external database required) and scales to PostgreSQL/pgvector for larger deployments. The database is the source of truth — FAISS is an optional read-acceleration sidecar, not a primary store.

Installation

pip install omop-emb                         # sqlite-vec backend (default, no extras needed)
pip install "omop-emb[pgvector]"             # adds PostgreSQL/pgvector support
pip install "omop-emb[faiss-cpu]"            # adds FAISS sidecar support
pip install "omop-emb[pgvector,faiss-cpu]"   # everything

Quick start

Ingest concepts (sqlite-vec, no external service):

export OMOP_EMB_BACKEND=sqlitevec
export OMOP_EMB_SQLITE_PATH=/data/omop_emb.db
export OMOP_CDM_DB_URL=postgresql+psycopg://user:pass@host:5432/omop_cdm

omop-emb embeddings add-embeddings --api-base http://localhost:11434/v1 --api-key ollama \
    --provider ollama --model nomic-embed-text:v1.5

Search:

omop-emb embeddings search --api-base http://localhost:11434/v1 --api-key ollama \
    --provider ollama --model nomic-embed-text:v1.5 \
    --query "hypertension" --query "type 2 diabetes" \
    --standard-only --domain Condition --k 5

Ingest concepts (OpenAI-hosted model):

Configure everything once via oa-configurator (see Configuration via oa-configurator below) rather than exporting connection details and passing an API key on the command line:

omop-config configure omop_alchemy
# CDM database connection

omop-config configure omop_emb
# backend: sqlitevec
# sqlite_path: /data/omop_emb.db
# provider_type: openai
# api_base: https://api.openai.com/v1
# api_key: <your OpenAI API key>
# embedding_model: text-embedding-3-large

omop-emb embeddings add-embeddings   # backend/CDM/provider/model/api_base/api_key all resolved from config

pgvector with HNSW index:

export OMOP_EMB_BACKEND=pgvector
export OMOP_EMB_DB_HOST=localhost
export OMOP_EMB_DB_USER=omop_emb
export OMOP_EMB_DB_PASSWORD=omop_emb
export OMOP_EMB_DB_NAME=omop_emb

omop-emb embeddings add-embeddings --api-base http://localhost:11434/v1 --api-key ollama \
    --provider ollama --model nomic-embed-text:v1.5
omop-emb maintenance rebuild-index --model nomic-embed-text:v1.5 --index-type hnsw --metric-type cosine

Environment variables

See the Configuration Reference for the complete list including asymmetric embedding prefixes and driver overrides.

Documentation

Full documentation: https://AustralianCancerDataNetwork.github.io/omop-emb

• Installation & backend setup
• Configuration reference
• Backend selection & index types
• CLI reference
• Interface guide

Roadmap

• sqlite-vec backend (default, zero-config)
• pgvector backend (PostgreSQL)
• HNSW index support for pgvector
• FAISS sidecar (approximate nearest-neighbour read acceleration)
• Embedding bundle export / import CLI (maintenance export, maintenance import, maintenance build-faiss-cache)
• In-DB concept filtering (domain, vocabulary, standard status, active status)
• Transparent FAISS fast path in EmbeddingReaderInterface
• Extensive backend and registry testing
• FAISS GPU support
• pgvectorscale support
• Vector quantisation for more efficient storage

Configuration via oa-configurator

The database connection can also be configured via oa-configurator, which stores settings in ~/.config/omop/config.toml and eliminates the need for environment variables at runtime:

omop-config init
omop-config configure omop_alchemy   # CDM database (required for ingestion)
omop-config configure omop_emb       # embedding database

omop-config configure omop_emb is required for local-dev setup before running the pgvector-backed test suite (CI provisions this automatically). Without it, those tests skip with "Resource 'test_emb_db' not configured" rather than failing.

See oa-configurator Setup for details.

Docker Compose

The included docker-compose.yaml provides both a CDM PostgreSQL database and a pgvector embedding database, plus a Python container with all optional backends pre-installed ([pgvector,faiss-cpu]). Default credentials work out of the box:

docker compose up

Include Ollama by adding the standalone profile:

docker compose --profile standalone up

The python-emb service runs omop-config configure at startup. To override credentials:

cp .env.example .env
docker compose up