dot-search 0.2.0

Augment existing database tables with vector and BM25 search

dot-search

dot-search augments existing database tables with vector search, BM25 keyword search, and exact substring matching. Results from multiple strategies are fused via Reciprocal Rank Fusion (RRF).

Prerequisites

PostgreSQL (production)

Requires pgvector and ParadeDB pg_search:

CREATE EXTENSION vector;
CREATE EXTENSION pg_search;

SQLite (testing only)

Uses sqlite-vec. No BM25 support.

Install

pip install dot-search

Environment variables

Variable	Example
DOT__EMBED_API_KEY	Bearer token
DOT__EMBED_BASE_URL	https://api.openai.com/v1
DOT__EMBED_MODEL	text-embedding-3-small
DOT__EMBED_DIMENSION	1536

Works with any OpenAI-compatible API (OpenAI, vLLM, TGI, Ollama, etc.).

Usage

import asyncio
from dot_search import SearchEngine, TableConfig, EmbeddingConfig, BM25Config, ExactConfig

engine = SearchEngine(db_url="postgresql+asyncpg://user:pass@localhost/mydb")

async def main():
    # --- 1. Index a table with vector search ---
    await engine.index(config=TableConfig(
        table="articles",
        embeddings=[
            EmbeddingConfig(source_column="body", target_column="ds_body"),
        ],
    ))

    # --- 2. Search ---
    results = await engine.search("neural networks", "articles", limit=5)
    for r in results:
        print(r.id, r.score)

    # --- 3. Re-index (config is persisted in ds_configs) ---
    await engine.index(index_id="articles")

    # --- 4. Add more embeddings, BM25, and exact search ---
    await engine.index(config=TableConfig(
        table="articles",
        embeddings=[
            EmbeddingConfig(source_column="body", target_column="ds_body"),
            EmbeddingConfig(source_column="title", target_column="ds_title"),
        ],
        bm25=[
            BM25Config(source_column="title"),
            BM25Config(source_column="body"),
        ],
        exact=[ExactConfig(source_column="name")],
    ))

    # --- 5. Hybrid search with SQL filter and weight overrides ---
    results = await engine.search(
        "fermentation and gut health",
        "articles",
        limit=10,
        where="published_year >= 2022 AND topic = 'health'",
        embedding_weights={"ds_body": 1.0, "ds_title": 0.3},
        bm25_weights={"title": 0.5, "body": 2.0},
    )

    # --- 6. Single-strategy search ---
    results = await engine.search("fermentation", "articles", strategy="bm25")
    results = await engine.search("Dupont", "articles", strategy="exact")

    # --- 7. Multiple indexes on the same table ---
    # Use case: a lightweight title-only index for autocomplete,
    # separate from the full hybrid index above.
    await engine.index(config=TableConfig(
        table="articles",
        index_id="article_titles",
        embeddings=[EmbeddingConfig(source_column="title", target_column="ds_article_titles_title")],
    ))
    # Search against the title-only index (uses its own config, not the "articles" one)
    results = await engine.search("gut health", "article_titles", limit=5)

asyncio.run(main())

Search strategies

Strategy	What it uses
"hybrid"	Vector + BM25 + exact (any configured), fused via RRF (default)
"vector"	Vector similarity only
"bm25"	BM25 keyword search only
"exact"	Substring (LIKE) search only

Contributing & Development

See docs/CONTRIBUTING.md and docs/DEVELOPMENT.md.

License

See LICENSE for details.

Contact

deepika Team — [e-mail] Project: gitlab.com/deepika6190303/deepika-open-toolbox/dot-search