doclighter 0.1.0

A semantic Ctrl+F that paints your document with a relevance gradient.

doclighter

A semantic Ctrl+F that paints your document with a relevance gradient.

doclighter is what you reach for when you need to see where a topic lives in a document — not be told an answer. It embeds your document at fine granularity, then projects query relevance back onto every word as a heatmap. No LLM, no hallucination, no top-K cliff.

from doclighter import Doclighter

doc = Doclighter.from_pdf("contract.pdf")
result = doc.search("termination clauses")
result  # In Jupyter: renders the whole document, color-coded by relevance

Why this exists

Traditional RAG hands an LLM the top 3–10 chunks and asks it to generate an answer. That's great when you trust the answer and don't need to read the source. But sometimes you need to read the source — legal review, paper skimming, contract diffing, due diligence — and you want a tool that helps you navigate a long document, not summarize it away.

doclighter is for that. It treats the whole document as the output, and re-colors it by semantic relevance to your query. Hot regions deserve your attention; cold regions you can skim past.

It's deterministic, fast (sub-100ms per query after indexing), and shows you the long tail — including the case where your query doesn't match anything (everything stays cold blue, which is itself useful information that RAG hides).

Install

pip install doclighter

Optional extras:

pip install "doclighter[quantize]"   # FAISS SQ8 index for very large docs
pip install "doclighter[streamlit]"  # for the interactive demo app
pip install "doclighter[dev]"        # for contributors

Quickstart

Load a document

from doclighter import Doclighter

# From a PDF on disk
doc = Doclighter.from_pdf("contract.pdf")

# From a PDF URL
doc = Doclighter.from_url("https://example.com/contract.pdf")

# From raw text
doc = Doclighter.from_text(open("paper.txt").read())

The first call downloads the default embedding model (~80 MB MiniLM) and embeds your document. For a ~10K word doc this takes ~25 seconds. Subsequent searches reuse the index.

Search

result = doc.search("termination clauses")

result.word_scores      # numpy array, shape (n_words,), values in [0, 1]
result.top_chunks(k=10) # list of (chunk_text, score, (start, end))
result.elapsed_ms       # ~10-50ms for typical docs
result.to_html()        # HTML string for display anywhere

In Jupyter, just put result on the last line of a cell — it renders the heatmap inline.

Zoom: the decay_sigma knob

The differentiating feature. decay_sigma controls how far semantic warmth spreads from a matched region:

narrow = doc.search("termination", decay_sigma=5.0)   # sharp word-level highlights
broad  = doc.search("termination", decay_sigma=80.0)  # broad thematic regions

Same index, no re-embedding. Drag the σ slider in the Streamlit demo to feel what this does.

Multi-query

result = doc.search(
    ["termination", "indemnification", "labour wages"],
    multi_query_aggregate="max",  # or "sum" to favor regions matching multiple
)

Save / load the index

Embedding is the slow step. Save once, reuse:

doc.save("contract.idx")
doc = Doclighter.load("contract.idx")

Bring your own embedder

Any callable mapping list[str] -> np.ndarray of shape (N, dim) works:

from sentence_transformers import SentenceTransformer

bge = SentenceTransformer("BAAI/bge-small-en-v1.5")
doc = Doclighter.from_text(text, embedder=bge.encode)

Streamlit demo

pip install "doclighter[streamlit]"
streamlit run examples/streamlit_app.py

A working UI with PDF upload, query box, σ slider, and live re-rendering.

How it works

1. Chunk the document into small rolling windows (default: 12 words, 50% overlap).

2. Embed each chunk with sentence-transformers (default: all-MiniLM-L6-v2).

3. Score each chunk against your query via cosine similarity.

4. Project chunk scores back onto every word via exponential proximity decay:

   word_score[w] = max over chunks c of  raw[c] × exp(-distance(w, c) / sigma)
   

5. Render the document as colored HTML — words inherit warmth from their nearest semantically matched chunk.

Step 4 is the interesting one. Max-aggregation (rather than sum) means a word's color reflects its single strongest semantic neighbor — visually intuitive and resistant to "many lukewarm chunks add up to red" noise.

How this compares to RAG

doclighter and RAG solve different problems:

	RAG	doclighter
Output	LLM-generated answer	Document, recolored
Best for	"What's the answer?"	"Where in the doc?"
When query has no answer	LLM hedges / hallucinates	Document stays cold (honest)
Hides chunk boundaries	No — top-K cliff	Yes — gradient smooths over
Cost per query	LLM tokens	Free (one matmul)
Determinism	Sampling-dependent	Fully deterministic

They're complementary. Use RAG when you trust the LLM with the question; use doclighter when you need to read the source yourself.

The algorithmic kernel is conceptually related to ColBERT's MaxSim late-interaction, but applied to visualization rather than ranking, and at the word-level rather than the token-level.

API reference

Doclighter(text, **kwargs)

Parameter	Default	Description
text	required	The document as a string
chunk_size	12	Words per rolling window
chunk_overlap	0.5	Fraction overlap between windows
embedder	None	Custom embedder callable (default: MiniLM)
embedding_model	"all-MiniLM-L6-v2"	sentence-transformers model name
decay_sigma	20.0	Default proximity decay scale (words)
quantize	False	Use SQ8 FAISS index instead of flat exact
quantize_rerank_k	200	If quantize=True, rerank top-K with exact

Alternate constructors: Doclighter.from_text(...), Doclighter.from_pdf(path, ...), Doclighter.from_url(url, ...).

doc.search(query, **kwargs) -> SearchResult

Parameter	Default	Description
query	required	A string, or list of strings for multi-query
decay_sigma	None	Override the doc's default sigma
multi_query_aggregate	"max"	"max", "mean", or "sum" for multi-query

SearchResult

Attribute / method	Returns
.word_scores	np.ndarray of shape (n_words,), in [0, 1]
.chunk_scores	np.ndarray of raw per-chunk cosine scores
.top_chunks(k=10)	list[(text, score, (start, end))]
.to_html(**kwargs)	HTML string of the heatmap-colored document
.elapsed_ms	Search latency

Development

git clone https://github.com/pratyush272/doclighter
cd doclighter
pip install -e ".[dev]"
pytest

License

MIT. See LICENSE.

Citation / acknowledgement

If you use doclighter in research, a link back is appreciated. The proximity-decay scoring idea borrows from passage-retrieval literature; max-aggregation over fine-grained matches is in spirit closest to ColBERT.