litesearch

Reading this on GitHub? The formatted documentation is easier to read.

litesearch adds search to an application without a search server. Your documents, their search indexes and their vectors live in one SQLite file, and every query runs two kinds of search over them and merges the results.

The first kind is keyword search. SQLite’s FTS5 extension keeps a full-text index, which lists for every word the rows that contain it. FTS5 ranks matching rows with bm25, a score that is higher when a row contains the query’s words more often and when those words are rare in the collection. Keyword search is exact. That is what you want for names, error codes and identifiers, and it finds nothing when a document says the same thing in different words.

The second kind is vector search. An encoder, also called an embedding model, turns a piece of text into a list of a few hundred numbers called an embedding. Texts with similar meanings get embeddings that are close together. litesearch stores an embedding beside each piece of text, embeds the query the same way, and returns the rows whose embeddings are closest. Vector search finds “ending a membership” when the user typed “cancel my subscription”, and it is weak at exact names.

Each kind of search returns a ranked list. litesearch merges the two lists with Reciprocal Rank Fusion (RRF): a row earns 1/(60 + its position) from each list it appears in, and the amounts add up. A row near the top of both lists beats a row at the top of only one. RRF uses positions only, so it never has to compare a bm25 score with a vector distance, and there is nothing to tune.

There are two ways to use the library.

Start with Index. It is built on database(), and Index.db is the database underneath, so you can run a lower-level query against the same data whenever Index does not do what you need.

Install

# usearch's SQLite extension is fetched, probed and configured on first use (see litesearch.postfix)
!uv add litesearch

litesearch has no optional extras. Two features need a package that is not installed with it: rerank=True needs flashrank, and the ONNX encoder FastEncode needs onnxruntime. Each is imported only when you use the feature, and the error message names the package to install. pip install litesearch installs everything else.

Route 1: Index

An Index is a searchable collection. add puts files, folders or text into it. Each document is cut into chunks, short passages of about 512 bytes, and each chunk is stored with its embedding. search returns the chunks that best match a query.

The other methods answer questions about whole documents and sections. sections ranks sections instead of chunks, read returns the full text of one section, toc lists everything in the index as a table of contents, and context collects the best sections for a query in a form you can pass to a language model.

ix = Index()                                        # pass a path to keep it on disk
ix.add('pdfs/attention_is_all_you_need.pdf')
hits = ix.search('how does multi-head attention work', limit=3)
[(h['heading'], h['page']) for h in hits]

Every hit has a heading and a node_id. While it ingests a document, Index builds a document tree from the document’s headings: the document is the root, its chapters and sections are the nodes below, and each chunk records the node it belongs to. heading is the path of headings above the chunk, such as attention is all you need › Abstract, and node_id names the section as the document’s id, #, and the section’s number. A hit then tells you which section matched, which is what a reader or a model needs, and not only which 512 bytes.

sec = ix.sections('how does multi-head attention work', limit=2)   # ranked *sections*, not chunks
[(s['node_id'], (s['snippets'] or [''])[0][:60]) for s in sec]

ix.read(sec[0]['node_id'])['text'][:300]            # one whole section, reassembled

ix.toc(summaries=False)                             # the corpus — no embeddings computed at all

One setting is left to you. rerank=True passes the top 30 candidates to a cross-encoder from flashrank, a small model that reads the query and each candidate together and scores how well they match. That is slower than comparing embeddings and more accurate. In the evaluation it improved weighted MRR by 0.026 to 0.077 and was positive in all twelve measured cells. It makes a query roughly 10 times slower and downloads a 4 MB model the first time.

ix.search('how does multi-head attention work', rerank=True)

For source code, use add_code. It cuts Python files at top-level functions and classes using Python’s own parser. Code chunks get no document tree, so they have no heading or node_id, and toc, sections and read do not cover them:

ix.add_code('litesearch')      # a directory, or the name of an installed package

Route 2: database()

database() opens a SQLite database and returns a fastlite Database. fastlite is a thin layer over SQLite that gives each table a Python object. The first vector search loads usearch’s SQLite extension into the connection, which adds vector distance functions you can call from SQL. db.usearch_sql() loads it before that, for SQL of your own. Pass a file path to keep the database on disk, or nothing for one held in memory.

db = database()
vecs = dict(v1=np.ones((100,), dtype=np.float32).tobytes(),
            v2=np.zeros((100,), dtype=np.float32).tobytes())
({m: db.q(f'select distance_{m}_f32(:v1,:v2) as d', vecs)[0]['d']
  for m in ['sqeuclidean', 'divergence', 'inner', 'cosine']}
 if db.usearch_sql() else 'usearch is off here, as on macOS 14')

There are four distance functions: cosine, sqeuclidean (squared Euclidean distance), inner (one minus the dot product) and divergence (Jensen-Shannon divergence). Each comes in four versions for the number type the vectors are stored in: f32, f16, f64 and i8. Smaller results mean closer vectors.

The cell below does by hand what Index does for you. It loads an encoder, creates a store (a table with a text column, an embedding column and a full-text index), embeds five sentences, and searches them.

The comment in the middle points at the one mistake that is easy to make. An embedding is stored as raw bytes, so a search has to be told the number type the bytes hold. litesearch assumes float16, which uses half the space of float32. The static encoder returns float32, so this code converts to float16 before storing. If you store float32 and search as float16, every distance comes back empty, litesearch prints a warning, and the vector half of the search returns rows in table order.

enc   = static_embedder()             # model2vec: no GPU, no ONNX runtime
store = db.get_store(hash=True, ann=True)

# float16, because that is what a store holds. Handing it float32 fails quietly: every distance
# comes back 0 and the ranking degrades to keyword-only with no error.
emb   = lambda xs: np.asarray(enc.encode(list(xs)), dtype=np.float16)

texts = ['attention mechanisms in neural networks', 'transformer architecture for sequences',
         'stochastic gradient descent and learning rate schedules',
         'positional encoding and token embeddings', 'dropout reduces overfitting']
store.insert_all([dict(content=t, embedding=e.tobytes()) for t, e in zip(texts, emb(texts))],
                 upsert=True, hash_id='id', hash_id_columns=['content'])
store.rebuild_index()

q = 'self-attention mechanism'
db.search(q, emb([q])[0].tobytes(), columns=['content'], limit=2)

Index exists so that this setup is written once, correctly.

Threads: share one connection, or give each thread its own

A SQLite connection used from several threads needs care. database(shared=True) wraps the connection so that threads can share it safely: each statement runs on its own, and a thread that opens a transaction keeps the connection until it commits. The default connection allows one thread at a time, so the alternative is to give each thread its own database() on the same file. SQLite’s WAL mode, which litesearch turns on, lets those connections read while another writes.

python -m evals.shared_conn measures both on a store of 20,000 rows with an HNSW index, on a 14-core arm64 Mac:

The last column needs explaining. By default SQLite counts every memory allocation under one lock shared by the whole process, so threads on separate connections still wait for each other. Running apsw.shutdown(); apsw.config(apsw.SQLITE_CONFIG_MEMSTATUS, 0); apsw.initialize() before the first connection opens turns that counting off. After that, apsw.memory_used() returns 0. litesearch does not do this for you, because it changes the whole process.

Share one connection when your threads already share one database object and writes are short. On a shared connection, a writer that commits continuously makes readers wait a median of 106 ms. For parallel reads, or a steady writer, give each thread its own connection and turn the counting off. evals/RESULTS.md has the method and the results for every thread count.

What the evaluation says

The defaults come from evals/. It runs 120 known-item queries per corpus over three corpora: EU legislation, arXiv papers and a 19th-century astrology treatise. A known-item query is one written to find a specific passage. The queries come in five styles, from a copy of the passage’s words to a paraphrase that shares none of them. The score is MRR (mean reciprocal rank: 1 if the right section is first, 1/2 if second, and so on, averaged over queries), counted per section and weighted so that three quarters of the weight falls on queries that do not copy the answer’s words. python -m evals.decide reproduces every number.

The changes above the blank row are on by default. The ones below it are off and stay off.

Three results matter more than any one row.

The choice of encoder matters little. Across potion-32M, bge-small, jina-v2-sm and egemma-300m the spread is 0.018 to 0.046. The static model, which looks up a vector per token instead of running a neural network, wins one corpus outright and indexes about 1,700 times more cheaply. The default is potion-multilingual-128M, a static model that also covers scripts other than Latin.

The document tree does not improve ranking, and it is still worth building. Ranking sections instead of chunks makes no difference. Assembling them does: on a Sanskrit corpus, context() nearly doubles verse-level recall over plain chunk search, from 0.190 to 0.340, the largest single effect in evals/.

Keyword search alone beats the hybrid search on this benchmark, and the benchmark is the reason. With pre(), keyword-only search beats hybrid in all 24 paired cells, because every query in the main set is a rewording of words from its target. evals/multihop.py builds a second set where the answer shares no word with the question. There keyword search cannot score at all, and the vector search finds the target at rank 1 between 53% and 84% of the time.

The other modules

Three related jobs live in separate packages. pdflite reads PDFs. ganapati handles Sanskrit metre, verse chunking and lemmas. vruksha builds an entity graph; it measured negative for retrieval and context no longer calls it (see evals/RESULTS.md).

Every store matches words across scripts, not only Sanskrit stores. A tokenizer is the part of FTS5 that splits text into the words it indexes. litesearch’s default tokenizer adds, beside each word, a copy of it reduced to plain ASCII letters, so श्रीमाता (Devanagari), śrīmātā (romanised with diacritics) and srimata all reach the same row. It also adds a second, looser copy that follows English spelling habits, so Bheeshma and Bhishma reach भीष्म and Bhīṣma. English words keep their own entries, so nothing that matched before stops matching. The looser copy makes the index 14.7% larger. This is the largest retrieval gain measured here: a Devanagari query finds the right verse every time (recall 1.000) with every encoder tested. The cost is that a database built with this tokenizer cannot be searched by a connection that has not registered it, which includes the plain sqlite3 command-line tool. Open it with database().

Next steps

• examples/01_simple_rag.ipynb ingests a folder of PDFs, chunks them with chonkie and reranks with FlashRank.
• examples/02_tool_use.ipynb gives litesearch to a language model as a tool.
• The api page covers Index and what each default is worth.
• The core page covers database, get_store, search, rrf_all, rank_fts, correct, fts_or and vec_search.
• The tree page covers add_dir, toc, read, sections and context.
• vishalakshi is a document vault built on Index.

Acknowledgements

Thank you to @yfedoseev for pdf-oxide, which litesearch uses to extract text from PDFs.