⚡️ dbs-vector

A High-Performance, Arrow-Native Local Codebase Search Engine and MCP Stdio Server for Apple Silicon.

dbs-vector is a optimized Retrieval-Augmented Generation (RAG) search engine designed specifically for macOS (M-Series chips). It bypasses traditional Python serialization bottlenecks by utilizing Apple's Unified Memory Architecture (UMA) and pure Apache Arrow data pipelines.

It enables lightning-fast, hybrid (Vector + Full-Text) search across your local codebase, entirely offline.

✨ Features

• Zero-Copy Memory Pipelines: Uses MLX to compute embeddings on the Mac GPU, casting the resulting tensors instantly into NumPy arrays via Unified Memory without costly float object instantiation.
• Arrow-Native Storage: Uses LanceDB to stream ingestion batches directly to disk via PyArrow, avoiding the massive memory overhead of JSON and dictionary comprehensions.
• Hybrid Retrieval: Simultaneously executes Approximate Nearest Neighbor (ANN) cosine vector search and native Tantivy Full-Text Search (FTS).
• Code-Aware Chunking: Intelligently splits documentation and code, respecting markdown fences so that code blocks are indexed as atomic units.
• Production Robustness: Features dynamic IVF_PQ indexing, Rust-level predicate pushdown (metadata filtering), and dataset compaction for delta-updates.
• Remote SQL API Ingestion: ApiChunker pulls pre-aggregated slow-query records from any networked backend over HTTP, replacing local files with a paginated REST API — no changes to the embedding or storage layers.
• Dynamic MCP Tools: dbs-vector mcp exposes one stdio MCP tool per configured engine from config.yaml; SQL engines additionally get an analytical browse_ tool and an impact-triage top_impacting_ tool for slow-query diagnosis.
• Directory Watch: document engines can re-ingest file changes from configured roots while dbs-vector mcp is running (uses the resident embedder). See docs/README_WATCH.md.

🚀 Installation

This project is built using uv, an extremely fast Python package manager.

1. Clone the repository:

   git clone https://github.com/dbsmedya/dbs-vector.git
   cd dbs-vector
   

2. Install the CLI package:

   uv sync
   

   This automatically sets up the environment and creates the dbs-vector executable in your path.

   Optional extras unlock additional ingestion sources:

   uv sync --extra sql  # DuckDB ingestion
   uv sync --extra api  # Remote HTTP API ingestion
   

💻 Usage

The application is entirely configuration-driven via config.yaml. It supports multiple data types (Engines) such as Markdown and SQL.

Global Options

• --config-file / -c: Path to your custom config.yaml (Defaults to ./config.yaml).

Ingesting Documents

Index markdown files, JSON SQL logs, DuckDB analytical files, or a remote HTTP slow-query API into the local vector store.

# Ingest all markdown files (default)
uv run dbs-vector ingest "docs/"

# Ingest every configured root for a document engine (paths: in config.yaml)
uv run dbs-vector ingest --type md

# Ingest SQL slow query logs (JSON format)
uv run dbs-vector ingest "slow_queries.json" --type sql

# Ingest SQL slow queries from DuckDB (High-Performance Columnar)
uv run dbs-vector ingest "slow_queries.duckdb" --type sql --rebuild

# Ingest from a remote HTTP API (paginated GET) — uses api_base_url from config.yaml
uv run dbs-vector ingest --type sql-api

# Or override the URL on the fly (without editing config.yaml):
uv run dbs-vector ingest "https://slow-log-api.internal/api/v1" --type sql-api

# Ingest via a custom SELECT sent to the remote API
uv run dbs-vector ingest "https://slow-log-api.internal/api/v1" --type sql-api \
  --query "SELECT fingerprint_id AS id, sanitized_sql AS text, db AS source, ..."

Searching the Codebase

Execute queries against your chosen engine.

# Semantic hybrid search across markdown
uv run dbs-vector search "What is MLX?"

# Limit results and restrict to a single source file/database
uv run dbs-vector search "What is MLX?" --limit 10 --source docs/architecture.md

# Find similar slow queries (SQL clustering)
uv run dbs-vector search "SELECT * FROM users" --type sql --min-time 1000

# Emit full results (score, source, full text, metadata) as JSON to stdout.
# Logs go to stderr, so stdout stays clean JSON and pipes safely into jq.
uv run dbs-vector search "SELECT * FROM users" --type sql --json | jq '.[].chunk.source'

Search options:

Indexes are built automatically at the end of every ingest run. Two indexes are created:

• IVF_PQ vector index (only when the table has > 256 rows)
• Tantivy FTS inverted index (required for hybrid search)

If you see a "Cannot perform full text search unless an INVERTED index has been created" error, it means the FTS index was never built for your table. Fix it by re-running ingestion — use --rebuild to wipe and re-index from scratch:

uv run dbs-vector ingest "docs/" --rebuild
uv run dbs-vector ingest "slow_queries.json" --type sql --rebuild

For detailed specifications on each ingestion source, see: 👉 SQL Engine Documentation 👉 DuckDB Ingestion Documentation 👉 Remote SQL API Ingestion

Model Context Protocol (MCP) Server

dbs-vector includes a built-in FastMCP server compatible with stdio-based MCP clients such as Claude Desktop and Claude Code.

# stdio — each client spawns its own dbs-vector process
uv run dbs-vector mcp

Each configured engine registers a tool named search_<engine_name> with dashes replaced by underscores, for example search_md, search_sql, and search_md_granite. Use the list_engines tool to inspect loaded engines, model contracts, profiles, and table names.

SQL engines get two additional analytical tools. browse_<engine> provides non-semantic access — point-lookup by id, ranking by a scalar column (calls, execution_time_ms) with no query string, and grouping/aggregation ("which table or user burns the most DB time"), plus derived impact_score and selectivity columns. top_impacting_<engine> is a one-call impact triage that returns the heaviest slow-query fingerprints ranked by impact_score = calls × execution_time_ms, each with a paste-ready exemplar SQL for EXPLAIN. Verbatim production SQL (raw_query, possibly PII) is exposed only when the server is launched with --allow-raw-queries (fail-closed by default). See the MCP documentation for the full per-tool reference.

For setup instructions, see: 👉 MCP Server Documentation

Bundled Claude Skills

dbs-vector ships two read-only SQL-performance skills under skills/ that combine its slow-query MCP tools with a live MySQL MCP (e.g. mysql-mcp-server):

• find-impacting-queries — finds and diagnoses the costliest queries: a one-call top_impacting_<engine> triage → live EXPLAIN + list_indexes
  • table_size → an index recommendation, an already-optimal / contention verdict, or a rewrite-candidate handoff. Index-first; no business knowledge required.
• query-rewrite — the handoff partner: rewrites a query when an index won't help (full-table aggregates, SELECT * + deep pagination, redundant joins, ORM noise), interviewing the domain owner first so a rewrite never silently changes results.

See MCP Server Documentation for the end-to-end workflow.

🏗 Architecture & Roadmap

dbs-vector is built upon strict Clean Architecture and SOLID principles. It utilizes a Configuration-Driven Registry Pattern, allowing new data engines (e.g., LibCST, Logs) to be added by simply updating config.yaml and registering new mappers/chunkers without modifying core orchestration logic.

Engines

See docs/README_EMBEDDINGS.md for model details.

Gemma vs Granite — which to use

Gemma engines (md, sql, sql-api) are the recommended default for most workloads: instruction-tuned with asymmetric search/clustering prefixes, fast on Apple Silicon, and consistently the strongest on English documentation. Reach for Granite engines (md-granite, sql-granite, sql-api-granite) when your corpus contains substantial non-English content (Granite R2 supports 200+ languages, Gemma 100+), when individual documents exceed Gemma's 2K-token context (Granite handles up to 32K), or when you want to A/B test chunk-size profiles against the Gemma baseline. Granite is a symmetric bi-encoder trained without instruction prefixes — leave passage_prefix and query_prefix empty when wiring a Granite engine. See docs/README_granite.md for tuning recipes and the rationale.

Specialized Gemma Workflows

The project is optimized for instruction-tuned models like embeddinggemma. It supports asymmetric task-based workflows defined in config.yaml:

• Markdown (Search Result): Uses the task: search result prefix for queries and title: none | text: for documents, maximizing retrieval accuracy for RAG.
• SQL (Clustering): Uses the task: clustering prefix for both ingestion and search, enabling high-precision semantic grouping of logically similar slow queries.

Future Hardware Support (CUDA/TPU)

Because the core RAG orchestration relies exclusively on the IEmbedder Protocol, the application is strictly hardware-agnostic at its core. While currently optimized for Apple Silicon via MLXEmbedder, future deployment to cloud GPUs or Linux environments simply requires implementing a new CudaEmbedder (using PyTorch/Transformers) that returns standard NumPy arrays. No changes to the ingestion, storage, CLI, or MCP layers are necessary to support new hardware accelerators. No access to a CUDA hardware at the moment.

For a deep dive into the engineering, the Apache Arrow ingestion lifecycle, and the blueprint for AST/LibCST integration, see the official documentation:

👉 Architecture & Engineering Documentation

🛠 Development

To contribute to dbs-vector, the project utilizes poethepoet as a task runner and implements strict quality gates (Ruff & Mypy).

# Run the entire validation suite (Format, Lint, Typecheck, Pytest)
uv run poe check

# Run tests with coverage
uv run poe test-cov