fabricatio-lancedb

MIT

LanceDB vector store backend for Fabricatio RAG. Provides async Rust-backed table operations — creation, document insertion, vector similarity search, and index rebuilding — plus Python-side RAG capability mixins and document models.

Architecture

The package has two layers:

The Rust layer manages LanceDB connections and tables. The Python layer implements the fabricatio-rag RAG interface on top of those primitives: batching embeddings, inserting documents, and searching by vector.

Installation

pip install fabricatio[lancedb]
# or
uv pip install fabricatio[lancedb]

For all Fabricatio extras:

pip install fabricatio[full]

Configuration

All options below are read through the fabricatio configuration chain (see the Configuration Guide at ../../docs/source/configuration.rst). Set them under the [ext.lancedb] table in fabricatio.toml, equivalently under [tool.fabricatio.ext.lancedb] in pyproject.toml, or via FABRICATIO_EXT__LANCEDB__<FIELD_UPPER> environment variables.

# fabricatio.toml
[ext.lancedb]
database_uri = "./lance.db"
default_table_name = "default"

Access at runtime: from fabricatio_lancedb.config import lancedb_config.

Usage

Low-level Rust API

Direct table operations with async Rust execution:

import asyncio
from fabricatio_lancedb.rust import VectorStoreService, VectorStoreTable, StoreDocument

async def main():
    # Connect to LanceDB
    service = await VectorStoreService.connect("./lance.db")

    # Create a table for 1536-dim embeddings (OpenAI text-embedding-3-small)
    table = await service.create_table("my_table", ndim=1536)

    # Add documents
    docs = [
        StoreDocument(
            content="LanceDB is a fast, open-source vector database.",
            vector=[0.1] * 1536,
        ),
        StoreDocument.with_metadata(
            content="Fabricatio is an LLM application framework.",
            vector=[0.2] * 1536,
            metadata={"source": "docs", "version": "1.0"},
        ),
    ]
    ids = await table.add_documents(docs)
    print(f"Inserted: {ids}")

    # Search by embedding
    results = await table.search_document(embedding=[0.15] * 1536, limit=5)
    for r in results:
        print(f"{r.id}: {r.content[:60]}...")
        print(f"  metadata: {r.access_metadata()}")

    # Rebuild the index after bulk inserts with rebuild_index=False
    await table.rebuild_index()

asyncio.run(main())

RAG capability (high-level)

Integrate with Fabricatio's RAG system using the LancedbRAG mixin:

from fabricatio_lancedb.capabilities.lancedb import LancedbRAG, LancedbAddRAGConfig, LancedbFetchRAGConfig
from fabricatio_lancedb.models.lancedb import LancedbDocumentModel

class MyDoc(LancedbDocumentModel):
    """Custom document with additional fields."""
    title: str = ""

class MyRAGRole(SomeBaseRole, LancedbRAG[MyDoc, LancedbAddRAGConfig, LancedbFetchRAGConfig[MyDoc]]):
    pass

async def run():
    role = MyRAGRole(name="my-rag-role")
    # Add a document — handles embedding batching internally
    await role.add_document(
        MyDoc(content="Semantic search with LanceDB and Fabricatio.", title="Intro"),
        config=LancedbAddRAGConfig(table_name="docs", embedding_batch_size=20),
    )
    # Fetch relevant documents
    results = await role.afetch_document(
        "how does semantic search work",
        config=LancedbFetchRAGConfig(document_model=MyDoc, limit=10),
    )
    for doc in results:
        print(f"[{doc.title}] {doc.content}")

Cached service helper

Reuse connections across calls:

from fabricatio_lancedb.inited_service import get_service

service = await get_service("s3://my-bucket/lancedb")
table = await service.open_table("production_docs")

API Reference

Rust layer (fabricatio_lancedb.rust)

VectorStoreService

VectorStoreTable

StoreDocument

Static constructor StoreDocument.with_metadata(content, vector, metadata: dict) serializes the metadata dict to JSON.

SearchedDocument

Method access_metadata() -> dict parses the JSON metadata into a Python dict.

Python layer

LancedbRAG

Extends RAG from fabricatio-rag with LanceDB storage.

LancedbAddRAGConfig

Dataclass config for add_document:

LancedbFetchRAGConfig

Dataclass config for afetch_document:

LancedbDocumentModel

Extends StoredDocumentModel and SearchedDocumentModel. Fields: content (str), metadata (dict | None).

Schema

Each LanceDB table uses this Arrow schema:

Dependencies

• fabricatio-core — core interfaces and configuration
• fabricatio-rag — base RAG abstractions (RAG, document models)
• more-itertools — chunked iteration for batch processing
• async-lru — async LRU caching

Rust dependencies (via PyO3): lancedb, arrow, pyo3, tokio.

License

This project is licensed under the MIT License — see LICENSE.