DuckDB Document Store for Haystack
A DuckDB-backed document store for Haystack with HNSW vector search via DuckDB's VSS extension. It supports:
- Dense embedding storage with HNSW indexing (cosine similarity, Euclidean distance, or inner product distance)
- Filtering with Haystack-style filter dictionaries
- In-memory operation or persistence via a DuckDB database file on disk
Installation (GitHub)
Use uv to install directly from the repository:
uv pip install "duckdb-haystack @ git+https://github.com/AdrianoKF/duckdb-haystack.git"
Usage
1) DocumentStore CRUD example
from haystack import Document
from haystack_integrations.document_stores.duckdb import DuckDBDocumentStore, document_store
store = DuckDBDocumentStore(
database=":memory:",
embedding_dim=3,
similarity_metric="cosine",
)
store.write_documents(
[
Document(id="doc-1", content="DuckDB is fast.", embedding=[0.1, 0.0, 0.9], meta={"source": "notes"}),
Document(id="doc-2", content="Haystack pipelines are modular.", embedding=[0.2, 0.1, 0.8]),
]
)
print("Total document count:", store.count_documents())
filters = {"field": "meta.source", "operator": "==", "value": "notes"}
filtered = store.filter_documents(filters=filters)
print("Filtered documents:", [doc.id for doc in filtered])
store.delete_documents(document_ids=["doc-2"])
print("After deletion:", store.filter_documents())
2) Retrieval with DuckDBRetriever in a pipeline
from haystack import Document, Pipeline
from haystack.components.embedders import SentenceTransformersDocumentEmbedder, SentenceTransformersTextEmbedder
from haystack_integrations.document_stores.duckdb import DuckDBDocumentStore
from haystack_integrations.retrievers.duckdb import DuckDBRetriever
store = DuckDBDocumentStore(database=":memory:", embedding_dim=384)
doc_embedder = SentenceTransformersDocumentEmbedder(model="sentence-transformers/all-MiniLM-L6-v2")
doc_embedder.warm_up()
documents = [
Document(content="DuckDB stores vectors in Float arrays backed by an HNSW index."),
Document(content="DuckDB is an analytical in-process SQL database management system."),
Document(content="Haystack offers composable pipelines."),
]
documents = doc_embedder.run(documents=documents)["documents"]
store.write_documents(documents)
query_embedder = SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2")
retriever = DuckDBRetriever(document_store=store)
pipeline = Pipeline()
pipeline.add_component("query_embedder", query_embedder)
pipeline.add_component("retriever", retriever)
pipeline.connect("query_embedder.embedding", "retriever.query_embedding")
result = pipeline.run(data={"query_embedder": {"text": "How does DuckDB store vectors?"}})
print(result["retriever"]["documents"][0].content)
License
duckdb-haystack is distributed under the terms of the Apache-2.0
license.
Metadata
Release files for duckdb-haystack 0.0.1.post1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| duckdb_haystack-0.0.1.post1.tar.gz | 125.6 kB | Without provenance | Details for duckdb_haystack-0.0.1.post1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| duckdb_haystack-0.0.1.post1-py3-none-any.whl | Python 3 | none | any | Details for duckdb_haystack-0.0.1.post1-py3-none-any.whl |
Total release size: 144.8 kB
Release files / duckdb_haystack-0.0.1.post1.tar.gz
| Download URL | duckdb_haystack-0.0.1.post1.tar.gz |
|---|---|
| Size | 125.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ffa377f41c82bccf1fba43240f8f9a08eb9d276a41f2a1b202644c6c07a416e5
|
|
BLAKE2b-256 checksum How to use checksums |
8f4a457acbef52297586758de3510098596e7964d0a46f9f01f86e3ea6d2040b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.9.26 {"installer":{"name":"uv","version":"0.9.26","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / duckdb_haystack-0.0.1.post1-py3-none-any.whl
| Download URL | duckdb_haystack-0.0.1.post1-py3-none-any.whl |
|---|---|
| Size | 19.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6d4697cf3c4e1ddcc95f93d56417e96d585b79d20bb7c8c153c1af5bd070447f
|
|
BLAKE2b-256 checksum How to use checksums |
9c1bf2e9adeaac101df3fd787090271420545baf0b9a6bc11277e78f95466a59
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.9.26 {"installer":{"name":"uv","version":"0.9.26","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for duckdb-haystack 0.0.1.post1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.