litesearch
NB Reading this on GitHub? The formatted documentation is nicer.
litesearch stores and searches documents in one SQLite file. FTS5 keyword search and SIMD vector search, fused by Reciprocal Rank Fusion. No server.
Two ways in. Pick by one question: do you want the defaults decided for you?
| route | use it when | what it costs |
|---|---|---|
[Index](https://Karthik777.github.io/litesearch/api.html#index) |
you want to search a folder of documents or code | nothing. Encoder, dtype, chunk size, retrieval and tree all come from evals/ |
[database()](https://Karthik777.github.io/litesearch/core.html#database) |
you need your own columns, encoder, SQL or float32 vectors | six decisions, one of which fails silently |
Start at Index. Drop to database() when it stops fitting: it is the same object underneath,
reachable as Index.db.
Install
# usearch SQLite extensions are configured automatically on first import
# (macOS needs one extra step — see litesearch.postfix)
!uv add litesearch
No extras. rerank=True wants flashrank, FastEncode wants onnxruntime, and each is imported
when used and says what to install. pip install litesearch gets the rest.
Route 1: Index
Six methods. add ingests, search returns chunks, sections returns sections, read opens
one, toc lists the corpus, context assembles an answer.
ix = Index() # pass a path to keep it on disk
ix.add('pdfs/attention_is_all_you_need.pdf')
hits = ix.search('how does multi-head attention work', limit=3)
[(h['heading'], h['page']) for h in hits]
Every hit carries a heading breadcrumb and a node_id, because Index builds a document tree
at ingest. That is what turns “which 512 characters” into “which section”.
sec = ix.sections('how does multi-head attention work', limit=2) # ranked *sections*, not chunks
[(s['node_id'], (s['snippets'] or [''])[0][:60]) for s in sec]
ix.read(sec[0]['node_id'])['text'][:300] # one whole section, reassembled
ix.toc(summaries=False) # the corpus — no embeddings computed at all
One knob is left to you. rerank=True runs a flashrank cross-encoder over the top 30 candidates:
+0.026 to +0.077 weighted MRR, positive in all twelve measured cells, at roughly 10x query latency
and a 4 MB download on first use.
ix.search('how does multi-head attention work', rerank=True)
For code, add_code uses the AST instead of headings, and its tree is module › class › function:
ix.add_code('litesearch') # a directory, or an installed package name
Route 2: database()
database() returns a fastlite Database patched with usearch’s
SIMD distance functions. Pass a path to persist, omit it for memory.
db = database()
vecs = dict(v1=np.ones((100,), dtype=np.float32).tobytes(),
v2=np.zeros((100,), dtype=np.float32).tobytes())
{m: db.q(f'select distance_{m}_f32(:v1,:v2) as d', vecs)[0]['d']
for m in ['sqeuclidean', 'divergence', 'inner', 'cosine']}
Four metrics, cosine, sqeuclidean, inner and divergence, each in f32, f16, f64 and
i8, running inside SQL.
Route 1 by hand is eight lines, and one of them is a trap:
enc = static_embedder() # model2vec: no GPU, no ONNX runtime
store = db.get_store(hash=True, ann=True)
# float16, because that is what a store holds. Handing it float32 fails quietly: every distance
# comes back 0 and the ranking degrades to keyword-only with no error.
emb = lambda xs: np.asarray(enc.encode(list(xs)), dtype=np.float16)
texts = ['attention mechanisms in neural networks', 'transformer architecture for sequences',
'stochastic gradient descent and learning rate schedules',
'positional encoding and token embeddings', 'dropout reduces overfitting']
store.insert_all([dict(content=t, embedding=e.tobytes()) for t, e in zip(texts, emb(texts))],
upsert=True, hash_id='id', hash_id_columns=['content'])
store.rebuild_index()
q = 'self-attention mechanism'
db.search(q, emb([q])[0].tobytes(), columns=['content'], limit=2)
Index exists because those eight lines have to be right every time.
What the evaluation says
evals/ runs 120 known-item queries per genre over three corpora (EU legislation, arXiv papers, a
19th-century astrology treatise) in five query flavours, scoring section-level MRR weighted so
three quarters of the mass sits where the query is not a copy of the answer.
python -m evals.decide reproduces every number.
Above the line is on by default. Below it is off and stays off.
| change | Δ weighted MRR | verdict |
|---|---|---|
pre() on the FTS leg |
+0.016 to +0.093 | on since 0.1.6 |
| 512-char chunks over page-sized | +0.06 to +0.12 | Index default |
| cross-encoder rerank | +0.026 to +0.077 | rerank=True, the one lever worth deciding |
| HNSW ANN vector leg | −0.005 | on by default; buy the speed |
| document tree, for ranking | −0.052 to +0.011 | a wash. Built for toc, read, sections |
| heading prefix on the chunk | ±0.02, sign flips by genre | a wash |
| deeper fanout alone | −0.014 to −0.068 | pays only with a reranker |
| late chunking | −0.033 to −0.053 | deleted; the code is in evals/latechunk.py |
| entity graph leg | −0.070 to −0.160 | vruksha, opt-in |
Three findings worth more than a table row.
The encoder is not the lever. Across potion-32M, bge-small, jina-v2-sm and egemma-300m
the spread is 0.018 to 0.046, and the static model wins one genre outright at ~1,700x cheaper
indexing. The default is potion-multilingual-128M, so non-Latin scripts are covered without
choosing an encoder.
The tree does not improve ranking and is still worth building. Section ranking is a wash.
Section assembly is not: on the Sanskrit corpus context() roughly doubles verse-level recall
over plain chunk search, 0.190 to 0.340, the largest single effect in evals/.
FTS alone looks unbeatable here, and that is the benchmark’s fault. Keyword-only retrieval
with pre() beats hybrid in all 24 paired cells, because every query in the main set is a lexical
transformation of its target. evals/multihop.py builds the corrective: a bridge set where the
answer shares no token with the question. There FTS cannot score at all and the vector leg reaches
the target at rank 1 between 53% and 84% of the time.
Beyond the two routes
| module | what you get |
|---|---|
litesearch.tree |
the tree directly: add_dir, doc_search, context, custom chunkers |
litesearch.data |
file_parse for any file, pyparse for code, FTS query preprocessing |
litesearch.utils |
encoders: static, ONNX FastEncode, image and multimodal |
litesearch.topics |
clusters and topic labels off the ANN index |
litesearch.sanskrit |
the cross-script FTS5 tokenizer |
litesearch.quality |
which documents in a store are retrieval noise |
Three things live in their own packages: pdflite reads PDFs, ganapati does Sanskrit metre, verse chunking and lemmas, and vruksha builds the entity graph.
Cross-script search is on for every store, not only Sanskrit ones. The sanskrit FTS5
tokenizer emits an ASCII fold of each token beside it, so श्रीमाता, śrīmātā and srimata all
reach the same row. Purely additive, ordinary English tokenises identically, and it is the largest
measured retrieval win here: 1.000 Devanagari to verse recall for every encoder tested. One cost:
a store built with this chain cannot be opened by a connection that has not registered the
tokenizer, plain sqlite3 included.
Next Steps
- examples/01_simple_rag.ipynb, ingest a folder of PDFs, chunk with chonkie, rerank with FlashRank
- examples/02_tool_use.ipynb, wire litesearch into an LLM tool-use loop
- api docs,
Index, and what each default is worth - core docs,
database,get_store,search,rrf_all,vec_search - tree docs,
add_dir,toc,read,sections,context - vishalakshi, a litesearch-backed vault, and the first caller nominated to port onto
Index; see the api page for what that port should test
Acknowledgements
A big thank you to @yfedoseev for pdf-oxide, which powers the PDF extraction functionality in litesearch.data.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file litesearch-0.1.34.tar.gz.
File metadata
- Download URL: litesearch-0.1.34.tar.gz
- Upload date:
- Size: 68.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.14.6
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
4566b0f6a9af51776c139b4eff4d0f78584d8df9901a9961d9d976e633a3f75b
|
|
| MD5 |
de35601688c031dfad2f7e01af8b904e
|
|
| BLAKE2b-256 |
416580e3b22458778c85e7bdd595c519d21cf3951a8c6ea42bf9fa6d1e96e1df
|
File details
Details for the file litesearch-0.1.34-py3-none-any.whl.
File metadata
- Download URL: litesearch-0.1.34-py3-none-any.whl
- Upload date:
- Size: 73.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.14.6
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
94e1896f4ab177099f4ac75d3be02ce866b34ec311f657c792fc3da42d01b61f
|
|
| MD5 |
91b8157832d3e20de1cfcc6795c3d67f
|
|
| BLAKE2b-256 |
ec02b3ffc5902cce2cf32183151e0254d5e7cfa75fda91c60b9676cb1303006f
|
