dlightrag 1.3.4

Multimodal RAG with knowledge graph and contextual intelligence

DlightRAG

Multimodal RAG with knowledge graph and contextual intelligence. Understands what your documents say, how concepts connect, and what the pages look like. Production-ready.

Most RAG systems treat documents as hierarchical text and search by similarity agentically — visual context is lost, entity relationships are missed, context filtering is limited. DlightRAG combines knowledge graph understanding with dynamic multimodal retrieval to close these gaps.

From text-heavy reports to chart-filled presentations — it adapts to your documents without information compromise. Inquiry answers come with inline citations grounded in actual document content. Flexibly ship it as a ready-to-run service, integrate into your backend, or expose as a tool for AI agents.

Features

• Dual multimodal RAG modes — Caption mode (parse → caption → embed) for pipeline-based multimodal paradigm; Unified mode (render → multimodal embed) for modern multimodal paradigm
• Knowledge graph + vector + visual retrieval — Multi-strategy retrieval across knowledge graph and vector similarity LightRAG, visual content, and dynamic metadata filters
• Multimodal ingestion — PDF, Images, Office Documents from local filesystem, Azure Blob Storage etc.
• Broad LLM support — Native SDKs for OpenAI, Anthropic, Gemini + any OpenAI-compatible endpoint
• Cross-workspace federation — Query across embedding-compatible workspaces with well managed merging
• Citation and highlighting — Inline citations with source, page, and highlighting attribution
• Observability — Zero-overhead telemetry via Langfuse for tracking pipelines, queries, and generations
• Four interfaces — Web UI, REST API, Python SDK, and MCP server

Architecture

_{Source: docs/architecture.drawio}

Quick Start

Defaults: qwen/qwen3.5-flash-02-23 (chat via OpenRouter) + voyage-multimodal-3.5 (embedding via Voyage) in unified mode. To use other providers or models, edit config.yaml — see Configuration.

Web UI

Click the image to watch demo (YouTube)

If you already have the REST API running (via Docker or dlightrag-api), the Web UI is available at:

http://localhost:8100/web/

Without Docker:

uv add dlightrag        # or: pip install dlightrag
cp .env.example .env    # set API keys in .env
dlightrag-api

Docker (Self-Hosted)

git clone https://github.com/hanlianlu/dlightrag.git && cd dlightrag
cp .env.example .env    # set API keys in .env; edit config.yaml for models/providers
docker compose up

Includes PostgreSQL (pgvector + AGE), REST API (:8100), and MCP server (:8101).

Local models (Ollama, Xinference, etc.): use host.docker.internal instead of localhost in base_url settings.

curl http://localhost:8100/health

curl -X POST http://localhost:8100/ingest \
  -H "Content-Type: application/json" \
  -d '{"source_type": "local", "path": "/app/dlightrag_storage/sources"}'

curl -X POST http://localhost:8100/retrieve \
  -H "Content-Type: application/json" \
  -d '{"query": "What are the key findings?"}'

curl -X POST http://localhost:8100/answer \
  -H "Content-Type: application/json" \
  -d '{"query": "What are the key findings?", "stream": true}'

Python SDK

uv add dlightrag        # or: pip install dlightrag
cp .env.example .env    # set API keys in .env

import asyncio
from dotenv import load_dotenv
from dlightrag import RAGServiceManager, DlightragConfig

load_dotenv()  # load .env

async def main():
    config = DlightragConfig()
    manager = RAGServiceManager(config)

    await manager.aingest(workspace="default", source_type="local", path="./docs")

    result = await manager.aretrieve(query="What are the key findings?")
    print(result.contexts)

    result = await manager.aanswer(query="What are the key findings?")
    print(result.answer)

asyncio.run(main())

Requires PostgreSQL with pgvector + AGE, or JSON fallback for development (see Configuration).

MCP Server (for AI Agents)

uv tool install dlightrag   # or: pip install dlightrag
cp .env.example .env        # set API keys in .env
dlightrag-mcp

{
  "mcpServers": {
    "dlightrag": {
      "command": "uvx",
      "args": ["dlightrag-mcp", "--env-file", "/absolute/path/to/.env"]
    }
  }
}

Tools: retrieve, answer, ingest, list_files, delete_files, list_workspaces — all with workspace isolation.

API & Internals

Method	Endpoint	Description
POST	/ingest	Ingest from local, Azure Blob, or Snowflake
POST	/retrieve	Contexts + sources (no LLM answer)
POST	/answer	LLM answer + contexts + sources (stream: true for SSE)
GET	/files	List ingested documents
DELETE	/files	Delete documents
GET	/api/files/{path}	Serve/download a file (local: stream, Azure: 302 SAS redirect)
POST	/reset	Reset workspace(s) — drop storage, clear indexes
GET	/workspaces	List available workspaces
GET	/health	Health check with storage status

All write endpoints accept optional workspace; read endpoints accept workspaces list for cross-workspace federated search. Set DLIGHTRAG_API_AUTH_TOKEN in .env to enable bearer auth.

• Request/response schema — docs/response-schema.md for ingestion parameters, retrieval contexts, sources, media, SSE streaming, citations, and multimodal queries.
• Retrieval & answer pipeline — docs/retrieval_answer_mechanism.md for unified vs caption mode, visual resolution, reranking, Step 1+2 merge.

Configuration

Configuration uses a hybrid system — structured app settings in config.yaml, secrets and deployment in .env.

Priority: constructor args > env vars > .env > config.yaml > defaults

See config.yaml for all application settings and .env.example for secrets/deployment reference.

Env var naming: all variables use the DLIGHTRAG_ prefix. Single underscore (_) is part of the field name (e.g. DLIGHTRAG_POSTGRES_HOST → postgres_host). Double underscore (__) means nested object (e.g. DLIGHTRAG_CHAT__MODEL → chat.model). See .env.example for details.

RAG Mode

The first decision — determines your ingestion pipeline, model requirements, and retrieval behavior.

Mode	Pipeline	Best for
caption	Document parsing → VLM captioning → text embedding → KG	Text-heavy documents, structured elements
unified (default)	Page rendering → multimodal embedding → VLM entity extraction → KG	Visually rich documents (charts, diagrams, complex layouts)

Caption mode parsers (parser in config.yaml):

Parser	Description
mineru (default)	MinerU PDF parser — fast, good for text-heavy documents
docling	Docling parser — alternative structure-aware parser
vlm	VLM-based OCR — renders pages and uses chat model (must be VLM) to extract structured content; no external parser dependency

All caption mode parsers use Docling's HybridChunker for structure-aware chunking.

Model usage by stage:

Stage	Caption	Unified
Image captioning	chat model (VLM)	chat model (VLM)
Table / equation captioning	chat model	—
Entity extraction	chat model	chat model (VLM)
Embedding	embedding model	embedding model (multimodal)
Rerank (llm_listwise)	ingest/chat model	chat model (VLM, pointwise scoring)
Rerank (API strategy)	cohere/jina/aliyun API	cohere/jina/aliyun API
Answer generation	chat model	chat model (VLM, sees text excerpts + page images)

Important: The chat model must support vision (multimodal/VLM). It doubles as the vision model for image captioning, VLM parser, unified mode, and multimodal queries. A text-only chat model will fail on these tasks.

For unified mode, set rag_mode: unified in config.yaml and use multimodal models:

# config.yaml
rag_mode: unified

chat:
  model: qwen3-vl-32b          # must support vision

embedding:
  model: Qwen3-VL-Embedding    # must be multimodal
  dim: 4096

Limitations: Snowflake is text-only (no visual embedding). A workspace is locked to one mode after first ingestion. Page images ~3-7 MB/page at 250 DPI.

Providers

Three native SDKs — choose per model block in config.yaml:

Provider	SDK	Use for
openai (default)	AsyncOpenAI	OpenAI, Azure OpenAI, Qwen/DashScope, MiniMax, Ollama, Xinference, OpenRouter, any OpenAI-compatible endpoint
anthropic	Anthropic SDK	Anthropic Claude models
gemini	Google GenAI SDK	Google Gemini models

Optional providers require extra dependencies:

pip install dlightrag[anthropic]   # Anthropic SDK
pip install dlightrag[gemini]      # Google GenAI SDK
pip install dlightrag[all]         # all optional providers

# config.yaml — OpenAI-compatible (Ollama example)
chat:
  provider: openai
  model: qwen3:8b
  base_url: http://localhost:11434/v1

# config.yaml — Anthropic (native SDK)
chat:
  provider: anthropic
  model: claude-sonnet-4-20250514

# config.yaml — Google Gemini (native SDK)
chat:
  provider: gemini
  model: gemini-2.5-pro

API keys go in .env:

DLIGHTRAG_CHAT__API_KEY=sk-...
DLIGHTRAG_EMBEDDING__API_KEY=sk-...

Storage Backends

Set in config.yaml:

Setting	Default	Options
vector_storage	PGVectorStorage	PGVectorStorage, MilvusVectorDBStorage, NanoVectorDBStorage, ...
graph_storage	PGGraphStorage	PGGraphStorage, Neo4JStorage, NetworkXStorage, ...
kv_storage	PGKVStorage	PGKVStorage, JsonKVStorage, RedisKVStorage, ...
doc_status_storage	PGDocStatusStorage	PGDocStatusStorage, JsonDocStatusStorage, ...

Note: When using PostgreSQL backends, LightRAG maps its internal namespace names to different table names (e.g. text_chunks → LIGHTRAG_DOC_CHUNKS, full_docs → LIGHTRAG_DOC_FULL). DlightRAG's unified mode adds a visual_chunks table via its own KV storage.

Workspaces

Each workspace has its own knowledge graph, vector store, and document index. workspace in config.yaml (default: default) is automatically bridged to backend-specific env vars — no manual setup needed.

Backend type	Isolation mechanism
PostgreSQL (PG*)	workspace column / graph name in same database
Neo4j / Memgraph	Label prefix
Milvus / Qdrant	Collection prefix
MongoDB / Redis	Collection scope
JSON / Nano / NetworkX / Faiss	Subdirectory under working_dir/<workspace>/

Reranking

Set in config.yaml under the rerank: block:

Setting	Default	Description
rerank.strategy	llm_listwise	llm_listwise, cohere, jina, aliyun, azure_cohere
rerank.model	(strategy default)	Model name sent to the endpoint
rerank.base_url	(provider default)	Custom endpoint URL for any compatible service
rerank.api_key	—	Set in .env as DLIGHTRAG_RERANK__API_KEY

Strategy	Default model	API key
llm_listwise	(uses ingest/chat model)	(reuses chat/ingest key)
cohere	rerank-v4.0-pro	DLIGHTRAG_RERANK__API_KEY
jina	jina-reranker-v3	DLIGHTRAG_RERANK__API_KEY
aliyun	qwen3-rerank	DLIGHTRAG_RERANK__API_KEY
azure_cohere	Cohere-rerank-v4.0-pro	DLIGHTRAG_RERANK__API_KEY

Point any backend at a local reranker (Xinference, etc.) via rerank.base_url + rerank.model in config.yaml.

Observability (Langfuse)

DlightRAG includes native, zero-overhead tracing using Langfuse. When configured, you get detailed waterfall traces of every RAG pipeline stage, LLM generation, and embedding call. If keys are omitted, the tracing module operates as a pure no-op with zero performance penalty.

To enable observability, set the following in your .env:

DLIGHTRAG_LANGFUSE_PUBLIC_KEY=pk-...
DLIGHTRAG_LANGFUSE_SECRET_KEY=sk-...
# DLIGHTRAG_LANGFUSE_HOST=https://cloud.langfuse.com  # Optional: defaults to cloud

This will automatically track retrieve, answer, and ingest operations at the service level.

Development

git clone https://github.com/hanlianlu/dlightrag.git && cd dlightrag
cp .env.example .env && uv sync
docker compose up -d                # PostgreSQL + API + MCP
docker compose up postgres -d       # PostgreSQL only

uv run pytest tests/unit            # unit tests (no external services)
uv run pytest tests/integration     # integration tests (requires PostgreSQL)
uv run ruff check src/ tests/ scripts/ --fix && uv run ruff format src/ tests/ scripts/

License

Apache License 2.0 — see LICENSE.

---

Built by HanlianLyu. Contributions welcome!