source-distiller 0.1.0

Zero-hallucination citation engine for multi-document research. NotebookLM-style grounding in your terminal.

Source Distiller

Zero-hallucination citation engine for multi-document research.
NotebookLM-style source grounding — in your terminal.

---

The Problem

You drop 15 research papers into an LLM and ask a question. It:

• Hallucinates citations that don't exist in any source
• Loses information buried in the middle of long context (Liu et al., 2023)
• Flattens real disagreements between sources into fake consensus
• Burns your entire token budget loading irrelevant pages

The Solution

Source Distiller is a deterministic, local-first CLI that sits between your documents and your LLM. It indexes sources with line-level anchors, retrieves only relevant evidence, detects cross-document conflicts, and audits every citation before it reaches the user.

No LLM calls. No cloud. No embeddings needed. No hallucinated references.

Real output from 15 academic papers (16,224 lines):

$ source-distiller index ./papers --out index.json
indexed 15 sources, 468 chunks -> index.json

$ source-distiller search --index index.json \
    --query "citation verification reduces hallucination" --top-k 3

[1] S4:L351-L390  score=19.842  source=CIVICA.pdf
    CIVICA achieved 92% citation precision, 91% citation coverage,
    and an 8% hallucination rate. Relative to citation-prompted RAG,
    CIVICA improved citation precision from 79% to 92% and reduced
    hallucination rate from 14% to 8%.

[2] S4:L316-L355  score=19.492  source=CIVICA.pdf
    Compared CIVICA against LLM-only, vanilla RAG, and citation-
    prompted RAG baselines using the same base generator...

[3] S4:L36-L75    score=18.160  source=CIVICA.pdf
    Unsupported claims are removed or restated using verified
    evidence only. On 1,200 consumer-law queries...

$ source-distiller quote --index index.json --cite S4:L351-L390
S4:L351-L390 CIVICA.pdf
351: [p5:L13] response accessibility for non-expert users.
352: [p5:L14] End-to-End Latency: Total response time from query...
353: [p5:L16] D. Quantitative Results
354: [p5:L17] Table I summarizes the main results. CIVICA achieved 92%
355: [p5:L18] citation precision, 91% citation coverage, and an 8% halluci-
356: [p5:L19] nation rate...

$ source-distiller conflicts --index index.json --top-n 3
Found 3 potential cross-source conflicts:

--- Conflict #1 (score=1.764) ---
  S3:L1331-L1370 [ALCE_Gao_2023.pdf]
  vs
  S13:L246-L285 [RAG_Lewis_2020.pdf]
  similarity=0.161  negation_signals=6
  numeric: table -> {'22', '10'} vs {'1'}

$ source-distiller audit --index index.json --answer draft.md
citations_found: 12
bad_citations: 0
possibly_uncited_blocks: 0
numeric_mismatches: 0

Why Not Just Use...

Tool	What It Does	What Source Distiller Does Differently
NotebookLM	Cloud-only, Google-proprietary, no CLI	Local, open-source, terminal-native, any agent
RAGFlow / LangChain	Full RAG stack, needs vector DB + LLM	Zero-dependency CLI, no LLM calls, deterministic
Manual grep	No semantic ranking, no citation tracking	BM25 scoring + citation audit + conflict detection
Just paste into Claude	Token-hungry, mid-context information loss	Index once, retrieve top-k, verify citations

Features

Feature	Description
8 commands	index search quote audit conflicts report chat stats
Line-level citations	Every claim traced to S1:L10-L20 with page anchors for PDFs
Citation audit	Catches fake sources, out-of-range lines, uncited paragraphs, numeric mismatches
Conflict detection	Auto-detects cross-document disagreements via topic similarity + negation + numeric diff
Full reports	One-command markdown report: source map + evidence + conflict matrix
Interactive REPL	source-distiller chat for real-time search, quote, and conflict checks
Quote verification	Re-opens cited spans so you (or your agent) can verify before trusting
Token-efficient	Index once, retrieve only top-k chunks — no full-doc context needed
Multi-format	PDF (pdftotext + pypdf), DOCX, Markdown, HTML, code, CSV, JSON, YAML
Zero core deps	Runs on Python stdlib. Optional pypdf for PDF fallback
Agent-ready	Built for Claude Code, Cursor, ChatGPT, OpenAI Codex CLI

Quick Start

pip install source-distiller

# Index your sources
source-distiller index ./my-papers --out index.json

# Search with a question
source-distiller search --index index.json --query "does retrieval reduce hallucination"

# Verify a citation before trusting it
source-distiller quote --index index.json --cite S4:L351-L390

# Detect conflicts across your sources
source-distiller conflicts --index index.json

# Generate a full evidence report
source-distiller report --index index.json --query "citation methods" --out report.md

# Audit a draft answer for bad citations
source-distiller audit --index index.json --answer draft.md

# Interactive mode
source-distiller chat --index index.json

Install from source

git clone https://github.com/emiroktay1/source-distiller.git
cd source-distiller
pip install -e ".[pdf]"

How It Works

 Documents (PDF, MD, DOCX, HTML, code...)
        |
        v
 +--------------+
 |    INDEX      |  Deterministic chunking with line anchors + bigram terms
 +--------------+  BM25-style TF-IDF scoring, no embeddings needed
        |
        v
 +--------------+
 |    SEARCH     |  Top-k retrieval with source ID + line citations
 +--------------+
        |
   +----+----+
   v         v
 +--------+ +------------+
 | QUOTE  | | CONFLICTS  |  Re-open cited span     Auto-detect cross-doc
 +--------+ +------------+  to verify support      disagreements
   |         |
   v         v
 +--------------+
 |    AUDIT     |  Catch fake citations, uncited claims, numeric drift
 +--------------+
        |
        v
 +--------------+
 |    REPORT    |  Source map + evidence + conflict matrix in markdown
 +--------------+

Interactive Mode

$ source-distiller chat --index index.json

Source Distiller v0.1.0 — Interactive Mode
Index: 15 sources, 468 chunks
Commands: /search <query> | /quote <S1:L10-L20> | /sources | /conflicts | /help | /quit

sd> citation verification hallucination
  [1] S4:L351-L390 score=19.8 CIVICA.pdf
      CIVICA achieved 92% citation precision, 91% citation coverage...
  [2] S4:L316-L355 score=19.5 CIVICA.pdf
      Compared CIVICA against LLM-only, vanilla RAG...

sd> /quote S4:L351-L390
  S4:L351-L390 CIVICA.pdf
  351: [p5:L13] response accessibility for non-expert users.
  352: [p5:L17] Table I summarizes the main results...

sd> /conflicts
  #1 (score=1.764) S3:L1331-L1370 vs S13:L246-L285
     numeric: table -> {'22', '10'} vs {'1'}

Agent Integration

Source Distiller is designed as a grounding layer for AI agents. It gives any LLM tool a way to cite, verify, and audit claims against real sources.

Claude Code / Codex CLI

# Add to your SKILL.md or agent prompt:
source-distiller index ./sources --out index.json
source-distiller search --index index.json --query "$USER_QUESTION" --json
source-distiller quote --index index.json --cite S1:L10-L25
source-distiller audit --index index.json --answer answer.md

Cursor / AI IDE

Add to .cursorrules:

When answering from documents, use source-distiller CLI to:
1. Index sources once
2. Search for relevant evidence with --json flag
3. Quote-check every citation before including it
4. Run audit on the final answer before presenting it

OpenAI Codex agent

# agents/openai.yaml
interface:
  display_name: "Source Distiller"
  default_prompt: "Use source-distiller to index sources, search evidence, cite every claim, and audit before answering."

Research Foundation

This tool operationalizes findings from peer-reviewed research:

Paper	Key Insight	How We Use It
Lost in the Middle (Liu 2023)	Models lose info buried in mid-context	Source mapping + top-k retrieval prevents positional bias
RAG (Lewis 2020)	Retrieval improves factuality	Index + search instead of dumping full docs into context
Self-RAG (Asai 2023)	Critique gates reduce hallucination	Quote-check and audit steps
ALCE (Gao 2023)	Citation presence != citation correctness	Audit catches fake/invalid citations
AIS (Rashkin 2023)	Attribution needs identified sources	Stable S1/S2/S3 source IDs with line anchors
CIVICA (2025)	Explicit verification > prompt-level citation	Verification/repair pipeline inspiration

Benchmark

python -m source_distiller.evaluate

Adversarial smoke-test with superseded policies, conflicting documents, fake citations, and semantic traps:

retrieval_score:    6/6  = 100%
quote_score:        1/1  = 100%
audit_score:        2/2  = 100%
conflicts_score:    1/1  = 100%
report_score:       1/1  = 100%
semantic_score:     1/1  = 100%
mechanical_total:  11/11 = 100%
overall_score:     12/12 = 100%

python -m pytest tests/ -v   # 18 tests, all passing

These are controlled smoke-tests, not claims of real-world perfection. See Limitations.

Output Shapes

Source Distiller defines three structured output templates for agents:

Shape	Use When	Sections
Full Source Synthesis	Research review, many sources, conflicts matter	Source Map + Evidence Ledger + Conflict Matrix + Answer + Caveats
Focused Answer	Direct question from many sources	Short Answer + Evidence + Conflicts/Gaps
Compatibility Audit	"Do these sources agree?"	Summary + Conflict Matrix + Canonical Recommendation

See docs/protocol.md for the complete protocol and checklist.

Limitations

• Lexical retrieval only. BM25-style scoring with bigrams. Paraphrases and synonyms may be missed. Use multiple query phrasings.
• No semantic entailment. Audit checks citation validity (exists, in range, numbers match) but cannot verify that a cited passage actually supports the claim. The agent must verify semantic support.
• PDF text quality. Depends on pdftotext or pypdf extraction. Scanned PDFs without OCR will produce empty output.
• Conflict detection is heuristic. Uses term overlap + negation signals + numeric comparison. Not a logical contradiction prover.

Roadmap

• Hybrid retrieval (lexical + lightweight embeddings)
• Semantic entailment scoring
• source-distiller watch — auto-reindex on file changes
• VS Code extension
• Web UI for non-terminal users

Contributing

See CONTRIBUTING.md. PRs welcome.

License

MIT