litesearch
Reading this on GitHub? The formatted documentation is easier to read.
litesearch adds search to an application without a search server. Your documents, their indexes and their vectors live in one SQLite file. Each query runs a keyword search and a vector search, and litesearch merges the two result lists.
| search | how it finds rows | good at | weak at |
|---|---|---|---|
| keyword | SQLite’s FTS5 extension keeps a full-text index: for each word, the rows that hold it. bm25 ranks the rows. A row scores higher when it holds the query’s words often and when those words are rare. | names, error codes, identifiers | the same idea in other words |
| vector | An encoder (an embedding model) turns text into an embedding, a list of a few hundred numbers. Texts with similar meanings get embeddings that are close together. | “ending a membership” for “cancel my subscription” | exact names |
Reciprocal Rank Fusion (RRF) merges the two lists. A row gets 1/(60 + its position) from each list that holds it, and the amounts add up. RRF uses positions only. It never compares a bm25 score with a vector distance, and it has no setting to tune.
There are two ways to use the library.
| route | use it when | what you decide |
|---|---|---|
[Index](https://Karthik777.github.io/litesearch/api.html#index) |
you want to search a folder of documents, a set of strings, or a code base | only whether to rerank. The encoder, vector type, chunk size, search strategy and document tree come from measurements in evals/ |
[database()](https://Karthik777.github.io/litesearch/core.html#database) |
you need your own columns, your own encoder, your own SQL, or float32 vectors | everything, including the vector dtype. A wrong dtype gives a wrong ranking and only a warning |
Start with Index. Index.db is the database under it. Use it when you need SQL against the same data.
Install
# usearch's SQLite extension is fetched, probed and configured on first use (see litesearch.postfix)
!uv add litesearch
litesearch has no optional extras. rerank=True needs flashrank. The ONNX encoder FastEncode needs onnxruntime. litesearch imports each one only when you use it, and the error names the package to install.
Route 1: Index
An Index is a searchable collection. add puts files, folders or text into it. Index cuts each document into chunks of about 512 bytes and stores each chunk with its embedding. search returns the chunks that best match a query.
Other methods work on whole documents and sections. sections ranks sections, read returns the full text of one section, toc lists the index as a table of contents, and context collects the best sections for a query, for a language model.
ix = Index() # pass a path to keep it on disk
ix.add('pdfs/attention_is_all_you_need.pdf')
hits = ix.search('how does multi-head attention work', limit=3)
[(h['heading'], h['page']) for h in hits]
Every hit has a heading and a node_id. During add, Index builds a document tree from the headings. The document is the root, its chapters and sections are the nodes, and each chunk records its node. heading is the path of headings above the chunk, such as attention is all you need › Abstract. node_id is the document id, #, and the section number. A hit tells you which section matched, not only which 512 bytes.
sec = ix.sections('how does multi-head attention work', limit=2) # ranked *sections*, not chunks
[(s['node_id'], (s['snippets'] or [''])[0][:60]) for s in sec]
ix.read(sec[0]['node_id'])['text'][:300] # one whole section, reassembled
ix.toc(summaries=False) # the corpus — no embeddings computed at all
You decide one setting. rerank=True gives the top 30 candidates to a cross-encoder from flashrank. A cross-encoder is a small model that reads the query and each candidate together and scores the match. In the evaluation it improved weighted MRR by 0.026 to 0.077, in all twelve measured cells. It makes a query about 10 times slower and downloads a 4 MB model on first use.
ix.search('how does multi-head attention work', rerank=True)
For source code, use add_code. It cuts Python files at top-level functions and classes with Python’s own parser. Code chunks get no document tree, so toc, sections and read do not cover them:
ix.add_code('litesearch') # a folder, or the name of an installed package
Route 2: database()
database() opens a SQLite database and returns a fastlite Database. fastlite is a thin layer over SQLite that gives each table a Python object. The first vector search loads usearch’s SQLite extension into the connection, which adds vector distance functions you can call from SQL. db.usearch_sql() loads it before that, for SQL of your own. Pass a file path to keep the database on disk, or nothing for one held in memory.
The cell below does by hand what Index does. It loads an encoder and creates a store: a table with a text column, an embedding column and a full-text index. Then it embeds five sentences and searches them.
Tell the search the number type of the stored vectors. An embedding is stored as raw bytes. litesearch assumes float16, which is half the size of float32. The static encoder returns float32, so the code converts to float16 before it stores. If you store float32 and search as float16, every distance comes back empty, litesearch prints a warning, and the vector search returns rows in table order.
db = database()
enc = static_embedder() # model2vec: no GPU, no ONNX runtime
store = db.get_store(hash=True, ann=True)
emb = lambda xs: np.asarray(enc.encode(list(xs)), dtype=np.float16) # a store holds float16
texts = ['attention mechanisms in neural networks', 'transformer architecture for sequences',
'stochastic gradient descent and learning rate schedules',
'positional encoding and token embeddings', 'dropout reduces overfitting']
store.insert_all([dict(content=t, embedding=e.tobytes()) for t, e in zip(texts, emb(texts))],
upsert=True, hash_id='id', hash_id_columns=['content'])
store.rebuild_index()
q = 'self-attention mechanism'
db.search(q, emb([q])[0].tobytes(), columns=['content'], limit=2)
What is under database()
database() returns a fastlite Database. Four packages sit under it, and each one has one job.
| package | its job in litesearch |
|---|---|
| apsw | the SQLite driver. It holds the connection, registers the Python FTS5 tokenizers and loads extensions. |
| apswutils | the table layer over apsw. It creates tables, adds columns, inserts and upserts rows, turns on FTS5 with its triggers, and writes the SQL of a full-text search. |
| fastlite | the Database object you use: db.t.name for a table, db.q(sql) for a query |
| adhara | a SQLite extension in Rust: a native tokenizer, and an exact vector search on all cores |
litesearch adds the stores, the tokenizers, the search functions and the thread safety. Your code calls litesearch and fastlite. It does not call apsw, apswutils or adhara directly.
adhara
database() loads adhara into each connection when adhara is installed. adhara is a dependency of litesearch on Linux and macOS. It has no Windows wheel, so on Windows litesearch works as before. If adhara does not load, database() gives a warning and the search runs in Python.
No function signature changes. A search returns the same rows in the same order with or without adhara. These behaviours are new:
vec_search, the exact vector search, asks adhara for the rowids of the nearest rows. adhara compares the query with every vector in memory, on all cores.vec_searchthen reads only those rows in SQL and computes_distas before.Database.searchcallsvec_searchwhenann=Falseor whenwhereis set.vec_searchdoes not use adhara for a search withoffset,where_args, an embedding column other thanembedding, thedivergencemetric orfloat64vectors.- Rows at the same distance come in rowid order. Before, their order depended on the SQLite query plan.
SANSKRIT_TOKENIZEis'adhara', the native form of the cross-script tokenizer. Use it for a new FTS table. A table made with'sanskrit'keeps working.- Each connection has the SQL table function
adhara_hybrid. It runs a keyword search, a vector search and RRF in one call. The adhara README documents it.
The HNSW search (ann=True with no where) and the keyword search do not use adhara. New stores keep the tokenizer chain porter simplify casefold 1 sanskrit unicodewords. A Swift app can read these stores with adhara’s Swift package.
The table shows Vault.search from vishalakshi, which is built on Index. The vault holds 181,290 chunks, on a 14-core arm64 Mac. Each time is for 10 searches. The hits were the same with and without adhara.
| search | without adhara | with adhara |
|---|---|---|
no where: HNSW |
51 ms | 52 ms |
a where that most rows pass (3 document kinds) |
518 ms | 118 ms |
a where that few rows pass (1 document kind) |
102 ms | 130 ms |
A where that few rows pass is slower with adhara. SQLite reads only the matching rows through an index, and adhara compares the query with every vector.
Threads: share one connection, or give each thread its own
A SQLite connection used from several threads needs care. database(shared=True) wraps the connection so that threads can share it. Each statement runs alone, and a thread that opens a transaction keeps the connection until it commits. Without shared=True, give each thread its own database() on the same file. SQLite’s WAL mode, which litesearch turns on, lets those connections read while another writes.
python -m evals.shared_conn measures both on a store of 20,000 rows with an HNSW index, on a 14-core arm64 Mac:
| shared | one each | one each, memstatus off | |
|---|---|---|---|
| FTS queries/s, 8 threads | 608 | 441 | 6,820 |
| brute-force vector scans/s, 8 threads | 70 | 176 | 168 |
| HNSW queries/s, 8 threads | 4,705 | 2,748 | 2,587 |
| FTS p50 / p99, 4 readers beside a writer | 106 / 193 ms | 13.4 / 33.0 ms | 1.6 / 3.4 ms |
| resident memory, HNSW index loaded | 17 MB | 17 MB per thread | 17 MB per thread |
By default SQLite counts every memory allocation under one lock for the whole process. Threads on separate connections then wait for each other. Run apsw.shutdown(); apsw.config(apsw.SQLITE_CONFIG_MEMSTATUS, 0); apsw.initialize() before the first connection opens to turn the counting off. After that, apsw.memory_used() returns 0. litesearch does not do this, because it changes the whole process.
Share one connection when your threads share one database object and writes are short. For parallel reads, or a steady writer, give each thread its own connection and turn the counting off. evals/RESULTS.md has the method and the results for every thread count.
What the evaluation says
The defaults come from evals/. It runs 120 known-item queries for each of three corpora: EU legislation, arXiv papers and a 19th-century astrology treatise. A known-item query is written to find one passage. The score is MRR (mean reciprocal rank): 1 if the right section is first, 1/2 if second, and so on. Three quarters of the weight is on queries that do not copy the words of the answer. python -m evals.decide gives every number.
The changes above the blank row are on by default. The changes below it are off.
| change | change in weighted MRR | decision |
|---|---|---|
pre() on the keyword search: drop stopwords, match word prefixes, join the words with OR |
+0.016 to +0.093 | on since 0.1.6 |
| 512-byte chunks instead of whole pages | +0.06 to +0.12 | the Index default |
| cross-encoder rerank | +0.026 to +0.077 | rerank=True |
| HNSW index for the vector search | -0.005 | on by default, for its speed |
| document tree, used for ranking | -0.052 to +0.011 | off. The tree is for toc, read and sections |
| heading text added to each chunk | about ±0.02, sign depends on the corpus | off |
| more candidates without a reranker | -0.014 to -0.068 | off; it helps only with a reranker |
| late chunking | -0.033 to -0.053 | removed; the code is in evals/latechunk.py |
| entity graph | -0.070 to -0.160 | removed from context; it is in vruksha |
The choice of encoder matters little: across four encoders the spread is 0.018 to 0.046. The default, potion-multilingual-128M, is a static model. It looks up one vector for each token, and it indexes about 1,700 times more cheaply than a neural encoder.
The document tree does not improve ranking. It does help a language model: on a Sanskrit corpus, context() raises verse-level recall from 0.190 to 0.340.
On this benchmark, keyword search alone beats hybrid search, because each query rewords words from its target. On evals/multihop.py, where the answer shares no word with the question, keyword search finds nothing, and vector search puts the target first in 53% to 84% of queries.
The other modules
| module | what you get |
|---|---|
litesearch.tree |
the document tree directly: add_dir, doc_search, context, custom chunkers |
litesearch.data |
file_parse for any file, pyparse for Python code, and the helpers that turn a typed query into an FTS5 query |
litesearch.utils |
encoders: static, ONNX FastEncode, image and multimodal |
litesearch.topics |
groups of similar chunks and labels for them, read off the HNSW index |
litesearch.sanskrit |
the FTS5 tokenizer that matches words across scripts |
litesearch.quality |
which documents in a store only add noise to search results |
Three related jobs are in other packages. pdflite reads PDFs. ganapati handles Sanskrit metre, verse chunking and lemmas. vruksha builds an entity graph.
Every store matches words across scripts. A tokenizer is the part of FTS5 that splits text into the words it indexes. litesearch’s tokenizer adds a copy of each word in plain ASCII letters, so श्रीमाता, śrīmātā and srimata reach the same row. It also adds a looser copy that follows English spelling, so Bheeshma reaches भीष्म and Bhīṣma. English words keep their own entries. The looser copy makes the index 14.7% larger. With it, a Devanagari query found the right verse every time (recall 1.000) with every encoder tested. A connection that has not registered the tokenizer cannot search the store. That includes the sqlite3 command-line tool. Open the store with database().
Next steps
- examples/01_simple_rag.ipynb reads a folder of PDFs, chunks them with chonkie and reranks with FlashRank.
- examples/02_tool_use.ipynb gives litesearch to a language model as a tool.
- The api page covers
Indexand what each default is worth. - The core page covers
database,get_store,search,rrf_all,rank_fts,correct,fts_orandvec_search. - The tree page covers
add_dir,toc,read,sectionsandcontext. - adhara is the Rust extension under
database(), with its Python and Swift packages. - vishalakshi is a document vault built on
Index.
Acknowledgements
Thank you to @yfedoseev for pdf-oxide, which litesearch uses to extract text from PDFs.
1 maintainer
Metadata
1 maintainer
Release files for litesearch 0.1.61
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| litesearch-0.1.61.tar.gz | 91.4 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| litesearch-0.1.61-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 188.7 kB
Release files / litesearch-0.1.61.tar.gz
| Download URL | litesearch-0.1.61.tar.gz |
|---|---|
| Size | 91.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4cec536dd2ab46c6b87ad59bff9267b3e8da0d40777dc4f31be0d2266f21270a
|
|
BLAKE2b-256 checksum How to use checksums |
a8b336ad0ddc931231d0c290770a2fa4fadc5a56c88159d2854e14ca9f5220a7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.6
|
Release files / litesearch-0.1.61-py3-none-any.whl
| Download URL | litesearch-0.1.61-py3-none-any.whl |
|---|---|
| Size | 97.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d3144d40b16fd53602c93dc0fe8c2b1bb457ee4548b904764da4b2387b42d3d8
|
|
BLAKE2b-256 checksum How to use checksums |
2a57422dcbc04542b8d38c1e84fe53a0dcc6fdab0b99e59e2302a84bc81c9122
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.6
|
Security for litesearch 0.1.61
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
