parqdb 0.2.0rc1

English | 中文

Billion-scale embedded vector database built entirely on Parquet and Arrow.

Quick Start | Status | Documentation

ParqDB is an embedded vector database for larger-than-memory search and analytics on billion-scale multimodal data, with Parquet storage and Arrow-native execution.

Key Features

• Billion-scale search in bounded memory. Search 1B vectors at 90.3% recall with 63.05 ms median latency using just 2 CPU cores and 4 GB of memory.
• Everything is Parquet. Source data and vector indexes use standard Parquet rather than proprietary binary formats, making indexes easy to version, publish, and share across engines and applications.
• Multimodal data, SQL-native search. Vector search is expressed as relational operations, allowing the SQL optimizer to combine it with filters, joins, and aggregations in a single execution plan.
• Built for serving and analytics. Parallelize across queries for online throughput, or within a query for low-latency analytical and large-k search.
• Scale from one core to thousands. Run embedded on a single machine, then use the same Parquet index with Spark or StarRocks at cluster scale.

Quick Start

Install ParqDB:

python -m pip install parqdb

From a new working directory, build a source-encoded IVF index over the dataset included in the package and run a filtered vector query:

import parqdb

session = parqdb.connect("./parqdb-data")
session.register_parquet("documents", parqdb.datasets.uri("documents"))
documents = session.table("documents")

documents.create_index(
    "documents_embedding",
    column="embedding",
    key=["document_id"],
    config=parqdb.IVF(nlist=3),
)
documents.wait_for_index("documents_embedding")

query = (
    documents.search([0.2, 0.0], column="embedding")
    .where("tenant_id = 42 AND status = 'published'")
    .nprobes(3)
    .limit(3)
    .select(["document_id", "title", "category"])
)

print(session.collect(query).to_pylist())

Vector search remains relational rather than becoming a terminal service call. Compile it as a SQL subquery and compose it with the rest of the analysis:

session.register_parquet(
    "document_stats",
    parqdb.datasets.uri("document_stats"),
)
search_sql = session.to_sql(query)
summary = session.sql(f"""
    SELECT
        h.category,
        COUNT(*) AS matches,
        AVG(h._distance) AS avg_distance,
        MAX(s.popularity) AS max_popularity
    FROM ({search_sql}) AS h
    JOIN document_stats AS s USING (document_id)
    GROUP BY h.category
    ORDER BY h.category
""")
print(summary.to_pydict())

The packaged dataset makes this example self-contained. The getting-started guide covers persistent tables, existing indexes, query inspection, and source schema requirements.

Status

The first supported product surface is the embedded DataFusion runtime. The index specification remains independent of that runtime; distributed engine adapters are no longer bundled into the Python package.

See the local guide for installation and configuration.

The experimental HTTP server is documented in the server guide.

Documentation

• Getting started and Python examples
• Core concepts, architecture, and open index specification
• Python API and configuration, including the server guide
• Current limitations, troubleshooting, and roadmap

TEngineDB-V and ParqDB

TEngineDB-V: An OLAP-Native Vector Search System for Large-k Workloads at Tencent is Tencent's production system for large-k vector search. On a 10-billion-vector deployment, its deep integration with TEngineDB delivers up to a 52x speedup over the legacy system.

ParqDB shares the idea, not the implementation. It rebuilds table-native vector search around open index formats and existing SQL engines, aiming for TEngineDB-V-class performance without requiring a proprietary engine.

If you use ParqDB in your research, please cite our VLDB 2026 Industry Track paper:

@misc{wu2026tenginedbvolapnativevectorsearch,
  title         = {{TEngineDB-V}: An {OLAP}-Native Vector Search System for Large-$k$ Workloads at Tencent},
  author        = {Xufei Wu and Pengcheng Zhang and Yitong Song and Xiaobo Zhang and Anqi Liang and Kai Wang and Jijun Du and Yidi Xiong and Guangxu Cheng and Zhe Chen and Peng Chen and Guoliang Li and Xuanhe Zhou and Fan Wu},
  year          = {2026},
  eprint        = {2608.00650},
  archivePrefix = {arXiv},
  primaryClass  = {cs.DB},
  url           = {https://arxiv.org/abs/2608.00650},
}

Development

ParqDB's next phase is being designed in public. We welcome concrete use cases, benchmark results, design feedback, and implementation help:

• Narrow the product around an embedded vector lakehouse
• Improve storage-backed Parquet search and measure the online serving envelope
• Add an extensible index-family framework
• Design compute-storage separation and build a complete DuckLake workflow

If you are working on RAG, agent trajectory storage, Parquet performance, or embedded lakehouse systems, share your workload and requirements in the relevant issue. Comment before starting a large change so that scope and interfaces can be agreed on first.

ParqDB uses uv, Maturin, Cargo, and a small Makefile orchestration layer:

make sync
make develop
make check

See CONTRIBUTING.md for quality gates, fixtures, benchmarks, and contribution guidelines.

License

ParqDB's original code is available under the MIT License. Wheels include the vendored DataFusion Python binding under Apache-2.0; see the third-party notices.

ParqDB builds on work from LanceDB, DataFusion, DuckDB, StarRocks, Apache Spark, and Apache Iceberg, with gratitude to their contributors and communities.