litesearch
NB Reading this on GitHub? The formatted documentation is nicer.
litesearch stores and searches documents in one SQLite file. FTS5 keyword search and SIMD vector search, fused by Reciprocal Rank Fusion. No server.
Two ways in. Pick by one question: do you want the defaults decided for you?
| route | use it when | what it costs |
|---|---|---|
[Index](https://Karthik777.github.io/litesearch/api.html#index) |
you want to search a folder of documents or code | nothing. Encoder, dtype, chunk size, retrieval and tree all come from evals/ |
[database()](https://Karthik777.github.io/litesearch/core.html#database) |
you need your own columns, encoder, SQL or float32 vectors | six decisions, one of which fails silently |
Start at Index. Drop to database() when it stops fitting: it is the same object underneath,
reachable as Index.db.
Install
# usearch SQLite extensions are configured automatically on first import
# (macOS needs one extra step — see litesearch.postfix)
!uv add litesearch
No extras. rerank=True wants flashrank, FastEncode wants onnxruntime, and each is imported
when used and says what to install. pip install litesearch gets the rest.
Route 1: Index
Six methods. add ingests, search returns chunks, sections returns sections, read opens
one, toc lists the corpus, context assembles an answer.
ix = Index() # pass a path to keep it on disk
ix.add('pdfs/attention_is_all_you_need.pdf')
hits = ix.search('how does multi-head attention work', limit=3)
[(h['heading'], h['page']) for h in hits]
Every hit carries a heading breadcrumb and a node_id, because Index builds a document tree
at ingest. That is what turns “which 512 characters” into “which section”.
sec = ix.sections('how does multi-head attention work', limit=2) # ranked *sections*, not chunks
[(s['node_id'], (s['snippets'] or [''])[0][:60]) for s in sec]
ix.read(sec[0]['node_id'])['text'][:300] # one whole section, reassembled
ix.toc(summaries=False) # the corpus — no embeddings computed at all
One knob is left to you. rerank=True runs a flashrank cross-encoder over the top 30 candidates:
+0.026 to +0.077 weighted MRR, positive in all twelve measured cells, at roughly 10x query latency
and a 4 MB download on first use.
ix.search('how does multi-head attention work', rerank=True)
For code, add_code uses the AST instead of headings, and its tree is module › class › function:
ix.add_code('litesearch') # a directory, or an installed package name
Route 2: database()
database() returns a fastlite Database patched with usearch’s
SIMD distance functions. Pass a path to persist, omit it for memory.
db = database()
vecs = dict(v1=np.ones((100,), dtype=np.float32).tobytes(),
v2=np.zeros((100,), dtype=np.float32).tobytes())
{m: db.q(f'select distance_{m}_f32(:v1,:v2) as d', vecs)[0]['d']
for m in ['sqeuclidean', 'divergence', 'inner', 'cosine']}
Four metrics, cosine, sqeuclidean, inner and divergence, each in f32, f16, f64 and
i8, running inside SQL.
Route 1 by hand is eight lines, and one of them is a trap:
enc = static_embedder() # model2vec: no GPU, no ONNX runtime
store = db.get_store(hash=True, ann=True)
# float16, because that is what a store holds. Handing it float32 fails quietly: every distance
# comes back 0 and the ranking degrades to keyword-only with no error.
emb = lambda xs: np.asarray(enc.encode(list(xs)), dtype=np.float16)
texts = ['attention mechanisms in neural networks', 'transformer architecture for sequences',
'stochastic gradient descent and learning rate schedules',
'positional encoding and token embeddings', 'dropout reduces overfitting']
store.insert_all([dict(content=t, embedding=e.tobytes()) for t, e in zip(texts, emb(texts))],
upsert=True, hash_id='id', hash_id_columns=['content'])
store.rebuild_index()
q = 'self-attention mechanism'
db.search(q, emb([q])[0].tobytes(), columns=['content'], limit=2)
Index exists because those eight lines have to be right every time.
Threads: one connection or one each
database(shared=True) wraps the connection so threads can share it. Each statement runs alone on it, and a transaction keeps it until the commit. The default connection takes one thread at a time, so give each thread its own database() on the same WAL file. python -m evals.shared_conn measures both on a 20,000-row store with an HNSW index, on a 14-core arm64 Mac.
| shared | one each | one each, memstatus off | |
|---|---|---|---|
| FTS queries/s, 8 threads | 608 | 441 | 6,820 |
| brute-force vector scans/s, 8 threads | 70 | 176 | 168 |
| HNSW queries/s, 8 threads | 4,705 | 2,748 | 2,587 |
| FTS p50 / p99, 4 readers beside a writer | 106 / 193 ms | 13.4 / 33.0 ms | 1.6 / 3.4 ms |
| resident memory, HNSW index loaded | 17 MB | 17 MB per thread | 17 MB per thread |
With SQLite’s defaults, every allocation takes one process-wide statistics mutex, and threads on separate connections queue on it. Running apsw.shutdown(); apsw.config(apsw.SQLITE_CONFIG_MEMSTATUS, 0); apsw.initialize() before the first connection opens removes it. apsw.memory_used() then returns 0. litesearch does not set it.
Share one connection when threads already share one database object and writes are short. A writer that commits back to back on a shared connection makes readers wait a median 106 ms. Give each thread its own connection, with the statistics off, for parallel reads or a steady writer. evals/RESULTS.md has the method and every thread count.
What the evaluation says
evals/ runs 120 known-item queries per genre over three corpora (EU legislation, arXiv papers, a
19th-century astrology treatise) in five query flavours, scoring section-level MRR weighted so
three quarters of the mass sits where the query is not a copy of the answer.
python -m evals.decide reproduces every number.
Above the line is on by default. Below it is off and stays off.
| change | Δ weighted MRR | verdict |
|---|---|---|
pre() on the FTS leg |
+0.016 to +0.093 | on since 0.1.6 |
| 512-char chunks over page-sized | +0.06 to +0.12 | Index default |
| cross-encoder rerank | +0.026 to +0.077 | rerank=True, the one lever worth deciding |
| HNSW ANN vector leg | −0.005 | on by default; buy the speed |
| document tree, for ranking | −0.052 to +0.011 | a wash. Built for toc, read, sections |
| heading prefix on the chunk | ±0.02, sign flips by genre | a wash |
| deeper fanout alone | −0.014 to −0.068 | pays only with a reranker |
| late chunking | −0.033 to −0.053 | deleted; the code is in evals/latechunk.py |
| entity graph leg | −0.070 to −0.160 | removed from context; vruksha only, see evals/RESULTS.md |
Three findings worth more than a table row.
The encoder is not the lever. Across potion-32M, bge-small, jina-v2-sm and egemma-300m
the spread is 0.018 to 0.046, and the static model wins one genre outright at ~1,700x cheaper
indexing. The default is potion-multilingual-128M, so non-Latin scripts are covered without
choosing an encoder.
The tree does not improve ranking and is still worth building. Section ranking is a wash.
Section assembly is not: on the Sanskrit corpus context() roughly doubles verse-level recall
over plain chunk search, 0.190 to 0.340, the largest single effect in evals/.
FTS alone looks unbeatable here, and that is the benchmark’s fault. Keyword-only retrieval
with pre() beats hybrid in all 24 paired cells, because every query in the main set is a lexical
transformation of its target. evals/multihop.py builds the corrective: a bridge set where the
answer shares no token with the question. There FTS cannot score at all and the vector leg reaches
the target at rank 1 between 53% and 84% of the time.
Beyond the two routes
| module | what you get |
|---|---|
litesearch.tree |
the tree directly: add_dir, doc_search, context, custom chunkers |
litesearch.data |
file_parse for any file, pyparse for code, FTS query preprocessing |
litesearch.utils |
encoders: static, ONNX FastEncode, image and multimodal |
litesearch.topics |
clusters and topic labels off the ANN index |
litesearch.sanskrit |
the cross-script FTS5 tokenizer |
litesearch.quality |
which documents in a store are retrieval noise |
Three things live in their own packages: pdflite reads
PDFs, ganapati does Sanskrit metre, verse chunking and
lemmas, and vruksha builds an entity graph (external; it measures
negative for retrieval and is no longer wired into context, see evals/RESULTS.md).
Cross-script search is on for every store, not only Sanskrit ones. The sanskrit FTS5
tokenizer emits an ASCII fold of each token beside it, so श्रीमाता, śrīmātā and srimata all
reach the same row; a plain-English fold beside it lets Bheeshma and Bhishma reach भीष्म and
Bhīṣma too. Purely additive: English keeps its tokens, the second fold grows the index 14.7%, and it is the largest
measured retrieval win here: 1.000 Devanagari to verse recall for every encoder tested. One cost:
a store built with this chain cannot be opened by a connection that has not registered the
tokenizer, plain sqlite3 included.
Next Steps
- examples/01_simple_rag.ipynb, ingest a folder of PDFs, chunk with chonkie, rerank with FlashRank
- examples/02_tool_use.ipynb, wire litesearch into an LLM tool-use loop
- api docs,
Index, and what each default is worth - core docs,
database,get_store,search,rrf_all,rank_fts,correct,fts_or,vec_search - tree docs,
add_dir,toc,read,sections,context - vishalakshi, a litesearch-backed vault, and the first caller nominated to port onto
Index; see the api page for what that port should test
Acknowledgements
A big thank you to @yfedoseev for pdf-oxide, which powers the PDF extraction functionality in litesearch.data.
1 maintainer
Metadata
1 maintainer
Release files for litesearch 0.1.42
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| litesearch-0.1.42.tar.gz | 77.9 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| litesearch-0.1.42-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 161.4 kB
Release files / litesearch-0.1.42.tar.gz
| Download URL | litesearch-0.1.42.tar.gz |
|---|---|
| Size | 77.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6d21fff1b6a698a7c619ae505e55788e273c5f4d923ead6267084928ade66f07
|
|
BLAKE2b-256 checksum How to use checksums |
d7d419deda33bc44f8648d19675325e9125dffe8f86a3707d24afad44ff6bb39
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.6
|
Release files / litesearch-0.1.42-py3-none-any.whl
| Download URL | litesearch-0.1.42-py3-none-any.whl |
|---|---|
| Size | 83.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f7f55252c3a03314ef37c85e446ac00a1ae2597923b61050290972282d696cd1
|
|
BLAKE2b-256 checksum How to use checksums |
12b5aa60393769351c2a5661d84ca640ba59cde14f09c36b40256ab5aca2b03a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.6
|
Security for litesearch 0.1.42
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
