BenoStreamDB Search (benostreamdb-search)

OpenSearch / Elasticsearch 7.10 and Qdrant REST compatibility gateway for BenoStreamDB.

⚠️ Security: This gateway ships stateless authentication (API key and/or JWT — see Authentication) but no TLS. Bind to 127.0.0.1 or an internal interface and terminate TLS at a reverse proxy. Do not expose it directly to the public internet. See SECURITY.md.

Overview

benostreamdb-search enables drop-in integration with applications, vector libraries, and ingest pipelines that speak:

1. OpenSearch / Elasticsearch 7.10 REST API (/_bulk, /{index}/_doc, /{index}/_search, aggregations, index templates).
2. Qdrant REST API (/collections, /collections/{name}/points, search, scroll).

It runs as a standalone daemon (bsdb-search) backed directly by BenoStreamDB tables on local or cloud object storage.

Installation

Via Cargo

cargo install benostreamdb-search

Via Python / PyPI

pip install benostreamdb-search

Running the Server

# Start server listening on port 9200 (OpenSearch) and 6333 (Qdrant)
bsdb-search --bind 0.0.0.0 --port 9200 --storage-uri file:///data/benostreamdb

Configuration

All configuration is via environment variables (or the equivalent CLI flags). These variables are specific to this gateway; the core engine's variables are documented in docs/CONFIGURATION.md.

Server & storage

Authentication

The gateway supports stateless authentication — no user database to maintain. Identity is delegated to a shared secret or an external IdP's signing key. The same credentials protect both the OpenSearch/ES API and the Qdrant API.

Health endpoints (/_health, /_cluster/health, /_cat/health, /healthz, /readyz, /livez) and GET / are always public.

# API key
export BSDB_API_KEY="$(openssl rand -hex 32)"
curl -H "api-key: $BSDB_API_KEY" http://127.0.0.1:9200/_cat/indices

# JWT (HS256) — verify a token minted by your IdP
export BSDB_JWT_SECRET="shared-secret"
curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:9200/_search

Where to keep the secret: never in the repo or a world-readable env file. Use a secret manager (Vault, AWS/GCP/Azure Secrets Manager) or a 0600 file mounted at /run/secrets/ (Docker/K8s) or /etc/benostreamdb/secrets/, loaded via systemd LoadCredential=/EnvironmentFile= (mode 0600).

Catalog storage policy (access control)

The gateway is an untrusted, network-facing connector: reads and writes are governed by the authoritative catalog storage policy — a .benostream-policy.json object in the warehouse/bucket root (a local path or an object store such as s3://, resolved through the core object store).

• Schemas come from the backend — the gateway never creates one. A schema exists because an operator provisioned it and named it in the policy. Writing an index whose schema is not granted returns 403.
• Writes are allowed only to schemas listed in readwrite_schemas. There is no default scratch namespace (unlike the local MCP server): a server connector is read-only until an operator publishes a policy.
• Reads have a read-only floor — with no policy every schema is readable (read-only) — and, when a policy is present, its readwrite_schemas / readonly_schemas / immutable lists are an allowlist for reads too. A read of a schema the policy does not name is denied before the index's existence is even checked.
• Immutable system schemas (default, public, information_schema, sys, system) are never writable.
• Policy updates are read-modify-write with optimistic concurrency (CAS on object stores that support conditional PUTs).

Enforcement is identical across the OpenSearch API (_search, _count, _doc/{id}, _mapping, _bulk, aliases, index create/delete) and the Qdrant API (search / query / scroll / retrieve / count / recommend / discover plus every mutating route).

Example .benostream-policy.json at the warehouse root:

{
  "policy_version": "1.0",
  "readwrite_schemas": ["analytics", "reporting"],
  "readonly_schemas": ["archive"],
  "no_access_schemas": ["pii"],
  "default_write_schema": "analytics"
}

With this policy the gateway may write analytics.* and reporting.* indexes, may only read analytics, reporting, and archive, and denies everything else.

Operational note: a deployment that publishes no policy is read-only — the gateway logs a WARN at startup ("search gateway is READ-ONLY: no writable schema"). Provision the schema(s) and publish a policy before expecting writes. Set BSDB_SEARCH_REQUIRE_POLICY=1 to refuse to start rather than run read-only. See docs/CATALOG_USAGE.md and docs/SUPPORT_MATRIX.md for the full model.

Iceberg catalog integration

benostreamdb-search can automatically synchronize search indices with an external Apache Iceberg catalog (Snowflake / Apache Polaris, Project Nessie, AWS Glue, Hive Metastore, or Unity Catalog) so ingested documents immediately advance the catalog's current snapshot.

Note: The gateway also accepts the core-engine aliases BSDB_CATALOG_TYPE, BSDB_CATALOG_URL / BSDB_CATALOG_URI, BSDB_CATALOG_TOKEN, BSDB_CATALOG_CREDENTIAL, and BSDB_CATALOG_NAMESPACE as fallbacks when the corresponding BSDB_SEARCH_CATALOG_* variable is unset.

Snowflake Polaris setup: Set BSDB_SEARCH_CATALOG_TYPE=rest, BSDB_SEARCH_CATALOG_URL=https://<account>.snowflakecomputing.com/polaris/api/catalog/v1, and BSDB_SEARCH_CATALOG_CREDENTIAL=<client_id>:<client_secret>. Every bulk ingest via /_bulk or _doc automatically executes an Iceberg atomic swap against Snowflake Polaris.