White-Walker 0.6.0

Multimodal Agentic RAG with hierarchical tree indexing — index PDFs & Markdown into navigable tree structures, persist in Supabase, and query with LLM-powered retrieval.

🧊 White Walker

Multimodal Agentic RAG — Index PDFs & Markdown into hierarchical tree structures, persist in Supabase, and query with LLM-powered retrieval.

White Walker transforms documents into navigable tree indexes using LLM-based structural analysis, stores them in Supabase (or locally), and answers questions by intelligently traversing the tree to find relevant sections.

---

✨ Features

• 🌲 Hierarchical Tree Indexing — Automatically detects document structure (TOC, sections, subsections) and builds a navigable tree
• 🗄️ Supabase Persistence — Trees, raw pages, and metadata stored in PostgreSQL via Supabase
• 🤖 Agentic Retrieval — LLM navigates the tree to find relevant sections, then synthesizes answers with citations
• 📄 PDF & Markdown Support — Works with academic papers, technical docs, and any structured document
• ⚡ Rate-Limited API — Built-in token-bucket rate limiter for NVIDIA NIM API (38 RPM, configurable)
• 🔌 Pluggable LLM Backend — Default: NVIDIA NIM API, easily extensible

---

📦 Installation

pip install white-walker

---

🚀 Quick Start

1. Set Environment Variables

# .env file
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_KEY=your-anon-key
NVIDIA_API_KEY=your-nvidia-nim-api-key

2. Set Up Supabase Tables

Run the SQL in setup_supabase.sql in your Supabase SQL editor to create the required tables (rag_documents, rag_raw_pages, rag_tree_nodes).

3. Index & Query

from white_walker import WhiteWalkerClient

client = WhiteWalkerClient()

# Index a PDF
doc_id = client.index("paper.pdf")

# Query the indexed document
result = client.query(doc_id, "What is the main contribution of this paper?")

print(result["answer"])
print(result["citations"])
print(result["confidence"])  # "high", "medium", or "low"

---

🏗️ Architecture

PDF / Markdown
       │
       ▼
┌─────────────────┐
│   PDF Ingestor   │  Extracts text, detects TOC, builds hierarchical tree
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Tree Indexer    │  LLM-based section detection, title verification,
│                  │  page-number alignment, node summarization
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Supabase Store  │  Persists tree nodes, raw pages, document metadata
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Agent Retriever  │  LLM navigates tree → finds relevant nodes →
│                  │  fetches page content → synthesizes cited answer
└─────────────────┘

---

⚙️ Configuration

White Walker uses white_walker/config.yaml for defaults:

Parameter	Default	Description
model	moonshotai/kimi-k2.6	NVIDIA NIM model for indexing & retrieval
toc_check_page_num	20	Max pages to scan for table of contents
max_page_num_each_node	10	Max pages per tree node
max_token_num_each_node	20000	Max tokens per tree node

Override at initialization:

client = WhiteWalkerClient(model="meta/llama-3.1-70b-instruct")

---

🔑 Environment Variables

Variable	Required	Description
NVIDIA_API_KEY	✅	NVIDIA NIM API key for LLM calls
SUPABASE_URL	Optional	Supabase project URL (falls back to local storage)
SUPABASE_KEY	Optional	Supabase anon/service key

---

📊 Evaluation

White Walker includes a RAGAS-based evaluation pipeline for benchmarking against multimodal agentic RAG datasets. See evaluate_pipeline.py for details.

---

📄 License

MIT License — see LICENSE for details.

---

🙏 Acknowledgments

Built on the foundation of VectifyAI/PageIndex, re-architected for local processing, Supabase persistence, and pluggable LLM backends.