dot-search 0.1.1

Augment existing database tables with vector and BM25 search

dot-search

dot-search augments existing database tables with vector search, BM25 keyword search, and exact substring matching. Results from multiple strategies are fused via Reciprocal Rank Fusion (RRF).

Prerequisites

PostgreSQL (production)

Requires pgvector and ParadeDB pg_search:

CREATE EXTENSION vector;
CREATE EXTENSION pg_search;

SQLite (testing only)

Uses sqlite-vec. No BM25 support.

Install

pip install dot-search

Environment variables

Variable	Example
DOT__EMBED_API_KEY	Bearer token
DOT__EMBED_BASE_URL	https://api.openai.com/v1
DOT__EMBED_MODEL	text-embedding-3-small
DOT__EMBED_DIMENSION	1536

Works with any OpenAI-compatible API (OpenAI, vLLM, TGI, Ollama, etc.).

Minimal usage

import asyncio
from dot_search import SearchEngine

engine = SearchEngine(db_url="postgresql+asyncpg://user:pass@localhost/mydb")

async def main():
    # Index — auto-serializes all columns, config saved to ds_configs
    await engine.index(table="documents")

    # Search
    results = await engine.search("neural networks", "documents", limit=5)
    for r in results:
        print(r.id, r.score)

asyncio.run(main())

Config is persisted in ds_configs. After the first index(), re-index without re-providing config:

await engine.index(index_id="documents")

Power usage

import asyncio
from dot_search import SearchEngine, TableConfig, EmbeddingConfig, BM25Config, ExactConfig

engine = SearchEngine(db_url="postgresql+asyncpg://user:pass@localhost/mydb")

async def main():
    # --- Index: multiple embeddings + BM25 + exact ---
    await engine.index(config=TableConfig(
        table="articles",
        embeddings=[
            EmbeddingConfig(source_column="body", default_weight=1.0),
            EmbeddingConfig(source_column="title", default_weight=0.5),
            EmbeddingConfig(
                source_column=None,                    # serialize all columns into one vector
                target_column="ds_articles_all",       # required when source_column=None
                default_weight=0.3,
            ),
        ],
        bm25=[
            BM25Config(source_column="title", default_weight=0.8),
            BM25Config(source_column="body", default_weight=1.0),
        ],
        exact=[ExactConfig(source_column="name")],
    ))

    # --- Hybrid search (vector + BM25 + exact, fused via RRF) ---
    results = await engine.search(
        "fermentation and gut health",
        "articles",
        limit=10,
        where="published_year >= 2022 AND topic = 'health'",
        embedding_weights={
            "ds_articles_body_default": 1.0,
            "ds_articles_title_default": 0.3,
            "ds_articles_all": 0.1,
        },
        bm25_weights={"title": 0.5, "body": 2.0},
    )

    # --- BM25-only ---
    results = await engine.search("fermentation", "articles", strategy="bm25")

    # --- Exact substring only ---
    results = await engine.search("Dupont", "articles", strategy="exact")

    # --- Multiple indexes on the same table ---
    await engine.index(config=TableConfig(
        table="articles",
        index_id="article_titles",
        embeddings=[EmbeddingConfig(source_column="title")],
    ))
    results = await engine.search("gut health", "article_titles", limit=5)

asyncio.run(main())

Search strategies

Strategy	What it uses
"hybrid"	Vector + BM25 + exact (any configured), fused via RRF (default)
"vector"	Vector similarity only
"bm25"	BM25 keyword search only
"exact"	Substring (LIKE) search only

Contributing & Development

See docs/CONTRIBUTING.md and docs/DEVELOPMENT.md.

License

See LICENSE for details.

Contact

deepika Team — [e-mail] Project: gitlab.com/deepika6190303/deepika-open-toolbox/dot-search