citadeldb-haystack

A Haystack DocumentStore backed by Citadel. Encrypted at rest, embedded in your process, and deletes that destroy the key, not just the row.

Passes deepset's own DocumentStoreBaseTests conformance suite.

pip install citadeldb-haystack

from haystack import Document
from haystack.utils import Secret
from citadeldb_haystack import CitadelDocumentStore

store = CitadelDocumentStore("corpus.cdl", Secret.from_env_var("CITADEL_KEY"))
# CITADEL_KEY must be set: an env-var secret is what lets a pipeline serialize.

store.write_documents([Document(id="d1", content="...", meta={"chapter": "intro"})])
store.filter_documents({"field": "meta.chapter", "operator": "==", "value": "intro"})

dim defaults to 768 and must match your embedding model.

The passphrase never lands in a pipeline file

The passphrase is a Haystack Secret. Pipelines are serialized to disk, and a literal token refuses to serialize, so a passphrase cannot be written into a pipeline by accident:

CitadelDocumentStore("literal.cdl", "literal-passphrase").to_dict()
# ValueError: Cannot serialize token-based secret.

CitadelDocumentStore("corpus.cdl", Secret.from_env_var("CITADEL_KEY")).to_dict()
# {... "key": {"type": "env_var", "env_vars": ["CITADEL_KEY"], ...}}

Use Secret.from_env_var for any store that goes into a saved pipeline.

Deletes destroy the key

Every document is sealed under its own key. Deleting destroys that key and then removes the row, so any ciphertext surviving elsewhere stays unreadable.

store.delete_documents(["d1"])
store.delete_all()             # returns the number erased

DuplicatePolicy.NONE falls back to FAIL, as InMemoryDocumentStore does, so an accidental re-write is reported rather than silently replacing a document whose key would then be destroyed.

Retrieval

query_embedding = [0.0] * 768        # from your Haystack text embedder, `dim` wide

store.embedding_retrieval(query_embedding, top_k=5,
                          filters={"field": "meta.chapter", "operator": "==", "value": "intro"})

Filtering uses Haystack's own evaluator, so the whole filter language, date comparisons included, matches InMemoryDocumentStore operator for operator. top_k is top_k: a filter matching only distant documents still returns them, however many others outrank them.

A top-level AND of string equality conditions is pushed into the scan, including nested paths like meta.person.name. Everything else is evaluated afterwards, so the two agree: nothing is pushed under OR or NOT, and numbers are not pushed either, because == here is Python's (1 == 1.0) where the stored comparison is JSON-type exact.

Notes

Documents Haystack did not embed are stored, filterable and countable, but take no part in embedding_retrieval, as in InMemoryDocumentStore, which retrieves only documents that have embeddings.

Citadel is embedded and one process owns the file. A path already open on this thread, under the same passphrase, is shared, so this can sit on the same database as another Citadel adapter; construct them on the same thread.

License

Apache-2.0