qdrant-lens 0.1.14

A knowledge browser and audit layer for Qdrant vector databases

qdrant-lens

A knowledge browser and audit layer for Qdrant vector databases.

Qdrant's own dashboard is a point browser — it shows raw vectors and payloads. qdrant-lens sits above it: it shows developers what the team has actually ingested, organised by source and document type, with a full audit trail, schema enforcement, and an LLM-powered knowledge layer (OKF). It is a drop-in wrapper around QdrantClient — existing code requires no changes.

---

Features

• Knowledge tree — browse ingested content by source → document type → document → chunk, not by raw vector ID; click any chunk to inspect its full _lens metadata
• Audit layer — every upsert and delete is intercepted, signed with the actor identity and timestamp, and written to an append-only sidestore (SQLite or Postgres); nothing in the cluster changes without a trace
• _lens schema enforcement — every point carries a validated metadata namespace; violations are surfaced before bad data reaches the index
• Sync-aware ingest — chunk-hash dedup means re-ingesting a folder only re-embeds what actually changed; unchanged files are skipped silently
• Hybrid search — dense + BM25 or SPLADE sparse vectors via FastEmbed; pass --bm25 at the CLI or bm25_enable=True on the client
• Parser-agnostic pipeline — LiteParse (default, Rust-core) handles PDFs, Office docs, images, and Jupyter notebooks; falls back to a pdfplumber/pytesseract stack automatically; .ipynb routed to a dedicated notebook parser
• Open Knowledge Format (OKF) — auto-generates .okf.md concept sidecars per file using a deepagents LLM pass; directory-level agent calls rebuild index.md with correct type grouping and handle orphan cleanup; lens okf CLI manages the knowledge base independently of ingest
• Dashboard UI — React frontend served at port 2873; tabs for Knowledge Tree, Upload, History, Analytics, Graph, Schedules, and Admin's Space; upload accepts drag-and-drop or Cmd/Ctrl+V clipboard paste; session persists across reloads via localStorage
• RBAC user management — four roles (viewer/developer/owner/admin), per-user bcrypt API keys, two-step provisioning via lens user bootstrap / create / activate; full audit log of all user management actions
• Knowledge Tree change detection — POST /tree/{collection}/diff compares a folder on disk against ingested doc_ids; sidebar shows VS Code-style green/yellow/red dots for new/modified/deleted files
• Chat agent — ask questions about a collection in plain English, backed by live tree/history/search tools; destructive operations require explicit typed confirmation
• Export — dump any collection to portable JSONL or JSON (vectors + metadata) with lens export

---

Install

pip install qdrant-lens                    # core: client wrapper + dashboard API
pip install "qdrant-lens[ingest]"          # + PDF/OCR/chunking pipeline (LiteParse included)
pip install "qdrant-lens[embed]"           # + FastEmbed (--embed-model CLI flag, sparse vectors)
pip install "qdrant-lens[chat]"            # + LangGraph chat agent + OKF generation
pip install "qdrant-lens[all]"             # everything

uv is recommended for projects: uv add "qdrant-lens[ingest,embed,chat]". See docs/quickstart.md for Docker and .env setup.

---

Quickstart

Day 1 — browse an existing collection (zero code changes)

pip install qdrant-lens
lens serve --url http://localhost:6333
# UI at http://localhost:2873
# Knowledge Tree renders from your existing collection immediately

Day 2 — drop-in client with audit log

from qdrant_lens import QdrantLensClient

# Drop-in replacement for QdrantClient — all existing calls unchanged
client = QdrantLensClient(
    url="http://localhost:6333",
    lens_actor="my-pipeline",   # labels every audit event
)

# Existing calls are intercepted, validated, and audited automatically
client.upsert(collection_name="docs", points=[...])

# New lens-only methods
tree       = client.tree("docs")
events     = client.history("docs", last_n=20)
violations = client.validate("docs")

Async variant:

from qdrant_lens import AsyncQdrantLensClient

client = AsyncQdrantLensClient(url="...", lens_actor="pipeline")
tree = await client.tree("docs")

Ingest a folder

# Basic — discover, parse, chunk, embed, upsert
lens ingest folder ./my-docs \
  --collection my-collection \
  --source github \
  --embed-model BAAI/bge-small-en-v1.5 \
  --actor ci-pipeline

# With BM25 hybrid search
lens ingest folder ./my-docs --collection my-collection --source local \
  --embed-model BAAI/bge-small-en-v1.5 --bm25

# With OKF knowledge sidecars (requires [chat] + LLM API key)
lens ingest folder ./my-docs --collection my-collection --source local \
  --embed-model BAAI/bge-small-en-v1.5 \
  --okf-generate --okf-model openai/gpt-4.1-mini

# In-memory Qdrant — test without a running server
lens ingest folder ./my-docs --collection test --source local \
  --embed-model BAAI/bge-small-en-v1.5 --memory

Or from Python:

from fastembed import TextEmbedding
from qdrant_lens import QdrantLensClient

client = QdrantLensClient(url="http://localhost:6333")
model  = TextEmbedding("BAAI/bge-small-en-v1.5")

results = client.ingest_folder(
    "./my-docs",
    collection_name="my-collection",
    source="github",
    embed_fn=lambda texts: [v.tolist() for v in model.embed(texts)],
    bm25_enable=True,
    okf_generate=True,
    okf_model="openai/gpt-4.1-mini",
)

---

CLI reference

Command	Description
lens serve	Start the dashboard API + UI at http://localhost:2873
lens collections	List all collections as a table
lens tree	Print the knowledge tree for a collection
lens history	Show the ingestion audit log for a collection
lens validate	Check _lens schema on all points; exit 1 on violations
lens export <collection>	Export points to JSONL or JSON (vectors + metadata)
lens init	Scaffold .lensignore, lens.yaml, and SETUP_GUIDE.md
lens migrate	Migrate unstructured points to _lens schema (Phase 2)
Ingest subcommands
lens ingest folder <path>	Ingest all supported files in a folder
lens ingest file <path>	Ingest a single file
lens ingest status	Show last ingest run summary from audit history
lens ingest list-models	List available FastEmbed embedding models
OKF subcommands
lens okf prune [folder]	Delete .okf.md sidecars whose source file no longer exists
lens okf fix-index [folder]	Rebuild index.md from existing sidecar frontmatter
lens okf update <file>	Refresh one sidecar's description, tags, and body
User subcommands
lens user bootstrap	Create the first admin user (run once on a fresh install)
lens user list	List all users, roles, and key prefixes
lens user create <username>	Create a pending user account
lens user activate <username>	Assign a role and generate an API key
lens user rotate-key <username>	Generate a new key, invalidating the old one
lens user set-role <username>	Change a user's role
lens user delete <username>	Deactivate a user account

Key flags for lens ingest folder:

Flag	Description
--collection / -c	Target collection name
--source / -s	Source label (e.g. github, gdrive)
--embed-model / -m	FastEmbed model (e.g. BAAI/bge-small-en-v1.5)
--actor	Actor label written to every audit event
--bm25	Enable BM25 sparse vectors
--sparse-model	Custom sparse model (overrides --bm25)
--okf-generate	Generate OKF sidecars via LLM
--okf-model	LiteLLM model string for OKF (e.g. openai/gpt-4.1-mini)
--memory	Use in-memory Qdrant — no server needed
--parser	liteparse (default) or auto
--chunk-size	Tokens per chunk (default 512)
--overlap	Token overlap between chunks (default 50)
--min-tokens	Minimum tokens per chunk; smaller chunks are merged (default 50)

---

Open Knowledge Format (OKF)

OKF auto-generates .okf.md YAML+markdown sidecars alongside each ingested file, acting as LLM-readable concept cards for RAG systems. Each sidecar contains a structured frontmatter block (type, title, description, tags, topics, entities, audience, key_questions, resource, timestamp) and a short prose summary.

During lens ingest folder, a single agent call per directory processes all added, updated, and deleted files in one pass and rebuilds index.md with correct type grouping. Orphaned sidecars (source file deleted) are detected and removed automatically.

Standalone management:

lens okf prune ./my-docs               # delete sidecars with no source file (--dry-run to preview)
lens okf fix-index ./my-docs/api/      # rebuild index.md from existing sidecar frontmatter
lens okf update ./my-docs/api/auth.py  # refresh one sidecar from current file content

Or via the Python API:

from qdrant_lens.ingest.okf_editor import OKFEditor

editor = OKFEditor(root_dir="./my-docs", model="openai/gpt-4.1-mini")
editor.prune_orphans(dry_run=True)          # list orphans without deleting
editor.fix_index()                           # rebuild all index.md files under root
editor.update_sidecar("./my-docs/api/auth.py")

See docs/okf.md for the full schema reference and deletion behaviour.

---

Dashboard UI

lens serve starts the FastAPI backend on port 2873 and serves the React frontend at /. Connect by entering the API URL and optional LENS_API_KEY on the connect screen — the header then shows the actor name from LENS_ACTOR.

Tab	What it shows
Knowledge Tree	source → doc_type → document → chunk hierarchy; Chunk Inspector; hover-to-delete; VS Code-style change indicators after running a diff scan
Upload	Drag-and-drop or Cmd/Ctrl+V paste files; configure collection, source, embed model; per-file results
History	Append-only audit log with time, action, actor, and chunk count
Analytics	Search query tracking and retrieval metrics
Graph	Document relationship graph (payload-derived edges: same_directory, same_tags, same_topic, same_entity; no Neo4j required)
Schedules	View and manage scheduled ingest jobs from lens.yaml
Admin's Space	User management (owner/admin only) — create users, generate keys, set roles, view full audit log

---

Environment variables

Variable	Default	Description
QDRANT_URL	http://localhost:6333	Qdrant instance URL
QDRANT_API_KEY	—	Qdrant API key
LENS_ADMIN_KEY	—	Permanent admin bypass key — always grants full access regardless of the users table; set this before exposing to a network
LENS_API_KEY	—	Legacy alias for LENS_ADMIN_KEY; honoured if LENS_ADMIN_KEY is not set
LENS_DB_URL	sqlite:///lens.db	Sidestore database (SQLite or Postgres URL)
LENS_ACTOR	pipeline	Actor name written to audit events in bootstrap mode only; once RBAC users exist, the authenticated username is used instead
LENS_CHAT_MODEL	openai/gpt-4.1-mini	LLM for the chat agent and OKF generation
OPENAI_API_KEY	—	Required for OpenAI-hosted OKF/chat models
NEO4J_URL	—	Neo4j bolt URL (Graph RAG only)
NEO4J_PASSWORD	—	Neo4j password (Graph RAG only)

---

Documentation

Guide	Description
Quickstart	Install, first run, Day 1/Day 2, env setup
Ingest Pipeline	File ingestion, parsers, chunking, dedup, hybrid search
Client Reference	QdrantLensClient API, audit log, schema, core concepts
Dashboard	lens serve, UI views, FastAPI routes, CLI commands
UI Guide	Dashboard walkthrough — Knowledge Tree, Upload, History, Analytics
OKF Reference	Open Knowledge Format — sidecars, editor, CLI, deletion behaviour
Export & Migration	Export collections to JSONL/JSON; migration recipes
Parsers	liteparse vs auto, system deps, troubleshooting
Testing	Test suite overview, per-module test descriptions
Contributing	Project structure, module map, adding parsers/routes/fields

---

Prerequisites

Requirement	Version	Notes
Python	>= 3.12
Qdrant	>= 1.18.0	Self-hosted, Docker, or Qdrant Cloud
Docker	any	For docker compose up (optional). Docker Hub login required — run docker login first, or use the Qdrant binary or brew install qdrant
Tesseract	any	OCR only — sudo apt install tesseract-ocr / brew install tesseract. Set TESSDATA_PREFIX if OCR fails at runtime
LibreOffice	any	Office → PDF conversion in the auto parser stack
Neo4j	>= 6.0	Graph RAG only — docker compose --profile graph up

---

Compatibility

• qdrant-client >= 1.18.0 required. Earlier versions are not supported.
• Python >= 3.12. 3.10 and 3.11 may work but are untested.

See COMPATIBILITY.md for the full version matrix.

---

Built with

Project	Role
Chonkie	Chunking — token-aware RecursiveChunker, SemanticChunker, and CodeChunker
LiteParse	Default parser — PDFs, Office docs, images without a LibreOffice dependency
PdfItDown	Office and image → PDF conversion in the auto fallback stack
FastEmbed	Dense and sparse (BM25/SPLADE) embeddings for the CLI and hybrid search
deepagents	Filesystem-scoped LLM agents powering OKF sidecar generation and editing
LangGraph	Chat agent orchestration for the POST /chat/{collection} route