litesearch 0.0.31

search through files with fts5, vectors and get reranked results. Fast

litesearch

NB Reading this on GitHub? The formatted documentation is nicer.

litesearch stores and searches documents in a single SQLite database. It combines FTS5 keyword search with SIMD vector similarity (via usearch), then merges the ranked lists using Reciprocal Rank Fusion. No server, no new infrastructure.

Module	What you get
litesearch (core)	database(), get_store(), db.search(), rrf_merge(), vec_search()
litesearch.data	PDF extraction, file parsing (file_parse), code indexing (pkg2chunks, dir2chunks), FTS query preprocessing
litesearch.utils	ONNX text, image, and multimodal encoders (FastEncode, FastEncodeImage, FastEncodeMultimodal)

Install

# usearch SQLite extensions are configured automatically on first import
# (macOS needs one extra step — see litesearch.postfix)
!uv add litesearch

error: Requirement name `litesearch` matches project name `litesearch`, but self-dependencies are not permitted without the `--dev` or `--optional` flags. If your project name (`litesearch`) is shadowing that of a third-party dependency, consider renaming the project.

Quick Start

Search your documents in eight lines of code:

db    = database()          # SQLite + usearch SIMD extensions loaded
store = db.get_store()      # table with FTS5 index + embedding column
enc = static_retrieval_embedder()
texts = ["attention is all you need",
         "transformers replaced recurrent networks",
         "gradient descent minimises the loss"]
embs  = enc.encode(texts)   # float32, shape (3, 512)
store.insert_all([dict(content=t, embedding=e.ravel().tobytes()) for t, e in zip(texts, embs)])

q = "self-attention mechanism"
db.search(q, enc.encode([q]).ravel().tobytes(), columns=['id','content'], dtype=np.float32, quote=True)

/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

[{'rowid': 1,
  'id': 1,
  'content': 'attention is all you need',
  '_dist': 0.7910182476043701,
  '_rrf_score': 0.016666666666666666},
 {'rowid': 3,
  'id': 3,
  'content': 'gradient descent minimises the loss',
  '_dist': 0.9670860767364502,
  '_rrf_score': 0.01639344262295082},
 {'rowid': 2,
  'id': 2,
  'content': 'transformers replaced recurrent networks',
  '_dist': 1.0227680206298828,
  '_rrf_score': 0.016129032258064516}]

Core API

database() — SQLite + SIMD

database() returns a fastlite Database patched with usearch’s SIMD distance functions. Pass a file path for persistence; omit it for an in-memory store.

db = database()   # ':memory:' by default; use database('my.db') for persistence
db.q('select sqlite_version() as sqlite_version')

[{'sqlite_version': '3.53.3'}]

The usearch extension adds SIMD-accelerated distance functions directly into SQL. Four metrics are available: cosine, sqeuclidean, inner, and divergence. All variants support f32, f16, f64, and i8 suffixes.

vecs = dict(
    v1=np.ones((100,),  dtype=np.float32).tobytes(),   # ones
    v2=np.zeros((100,), dtype=np.float32).tobytes(),   # zeros
    v3=np.full((100,), 0.25, dtype=np.float32).tobytes()  # 0.25s (same direction as v1)
)
def dist_q(metric):
    return db.q(f'''
        select
            distance_{metric}_f32(:v1,:v2) as {metric}_v1_v2,
            distance_{metric}_f32(:v1,:v3) as {metric}_v1_v3,
            distance_{metric}_f32(:v2,:v3) as {metric}_v2_v3
    ''', vecs)

for fn in ['sqeuclidean', 'divergence', 'inner', 'cosine']: print(dist_q(fn))

[{'sqeuclidean_v1_v2': 100.0, 'sqeuclidean_v1_v3': 56.25, 'sqeuclidean_v2_v3': 6.25}]
[{'divergence_v1_v2': 34.657352447509766, 'divergence_v1_v3': 12.046551704406738, 'divergence_v2_v3': 8.66433334350586}]
[{'inner_v1_v2': 1.0, 'inner_v1_v3': -24.0, 'inner_v2_v3': 1.0}]
[{'cosine_v1_v2': 1.0, 'cosine_v1_v3': 0.0, 'cosine_v2_v3': 1.0}]

Cosine distance between v1 (ones) and v3 (0.25s) is 0.0 — they point in the same direction. Both inner and divergence are also available for different retrieval trade-offs.

get_store() — FTS5 + Embedding Table

db.get_store() creates (or opens) a table with a content TEXT column, an embedding BLOB column, a JSON metadata column, and an FTS5 full-text index that stays in sync automatically via triggers.

store = db.get_store()   # idempotent — safe to call multiple times
store.schema

'CREATE TABLE [store] (\n   [content] TEXT NOT NULL,\n   [embedding] BLOB,\n   [metadata] TEXT,\n   [uploaded_at] FLOAT DEFAULT CURRENT_TIMESTAMP,\n   [id] INTEGER PRIMARY KEY\n)'

Pass hash=True to use a content-addressed id (SHA-1 of the content). Useful for code search and deduplication — re-inserting the same content is a no-op:

code_store = db.get_store(name='code', hash=True)
code_store.insert_all([
    dict(content='hello world', embedding=np.ones( (100,), dtype=np.float16).tobytes()),
    dict(content='hi there', embedding=np.full( (100,), 0.5, dtype=np.float16).tobytes()),
    dict(content='goodbye now', embedding=np.zeros((100,), dtype=np.float16).tobytes()),
], upsert=True, hash_id='id')
code_store(select='id,content')

[{'id': '250ce2bffa97ab21fa9ab2922d19993454a0cf28', 'content': 'hello world'},
 {'id': 'c89f43361891bfab9290bcebf182fa5978f89700', 'content': 'hi there'},
 {'id': '882293d5e5c3d3e04e8e0c4f7c01efba904d0932', 'content': 'goodbye now'}]

db.search() — Hybrid FTS + Vector with RRF

db.search() runs both an FTS5 keyword query and a vector similarity search, then merges the ranked lists with Reciprocal Rank Fusion. Documents that appear in both lists get a score boost — the best of both worlds.

# Re-create a clean store for the search demo
db2  = database()
st2  = db2.get_store()

phrases = [
    "attention mechanisms in neural networks",
    "transformer architecture for sequence modelling",
    "stochastic gradient descent and learning rate schedules",
    "positional encoding and token embeddings",
    "dropout regularisation reduces overfitting",
]
# use float32 vectors (matching dtype= below)
vecs2 = [np.random.default_rng(i).random(64, dtype=np.float32) for i in range(len(phrases))]
st2.insert_all([dict(content=p, embedding=v.tobytes()) for p, v in zip(phrases, vecs2)])

<Table store (content, embedding, metadata, uploaded_at, id)>

q2 = "attention"
q_vec = np.random.default_rng(42).random(64, dtype=np.float32).tobytes()
db2.search(q2, q_vec, columns=['id','content'], dtype=np.float32)

[{'rowid': 1,
  'id': 1,
  'content': 'attention mechanisms in neural networks',
  'rank': -1.116174474454989,
  '_rrf_score': 0.032539682539682535},
 {'rowid': 3,
  'id': 3,
  'content': 'stochastic gradient descent and learning rate schedules',
  '_dist': 0.20330411195755005,
  '_rrf_score': 0.016666666666666666},
 {'rowid': 2,
  'id': 2,
  'content': 'transformer architecture for sequence modelling',
  '_dist': 0.23124444484710693,
  '_rrf_score': 0.01639344262295082},
 {'rowid': 5,
  'id': 5,
  'content': 'dropout regularisation reduces overfitting',
  '_dist': 0.23238885402679443,
  '_rrf_score': 0.016129032258064516},
 {'rowid': 4,
  'id': 4,
  'content': 'positional encoding and token embeddings',
  '_dist': 0.32342469692230225,
  '_rrf_score': 0.015625}]

Pass rrf=False to see the raw FTS and vector legs separately — handy for debugging relevance:

db2.search(q2, q_vec, columns=['id','content'], dtype=np.float32, rrf=False)

{'fts': [{'id': 1,
   'content': 'attention mechanisms in neural networks',
   'rank': -1.116174474454989}],
 'vec': [{'id': 3,
   'content': 'stochastic gradient descent and learning rate schedules',
   '_dist': 0.20330411195755005},
  {'id': 2,
   'content': 'transformer architecture for sequence modelling',
   '_dist': 0.23124444484710693},
  {'id': 5,
   'content': 'dropout regularisation reduces overfitting',
   '_dist': 0.23238885402679443},
  {'id': 1,
   'content': 'attention mechanisms in neural networks',
   '_dist': 0.24136507511138916},
  {'id': 4,
   'content': 'positional encoding and token embeddings',
   '_dist': 0.32342469692230225}]}

Tip — dtype matters. Always pass the same dtype used when encoding. model2vec and most ONNX models return float32; pass dtype=np.float32. The default is float16 (matches FastEncode).

Tip — custom schemas. get_store() is a convenience. For custom schemas, call db.t['my_table'].vec_search(emb, ...) and rrf_merge(fts_results, vec_results) directly.

ANN store — HNSW for large corpora

The default vector search is an exact brute-force scan (distance_* over every row) — great up to a few thousand rows. Past that, pass ann=True to get_store() to also maintain a usearch HNSW index as a rebuildable sidecar file. The SQLite table stays the source of truth (content + embedding blob + FTS5); the index is a pure accelerator you can rebuild anytime with store.rebuild_index().

• store.sync(content, key_col=..., emb_fn=...) — smart hash-diff update: deletes stale rows, embeds + upserts changed ones, and mirrors both into the index.
• db.search(..., ann=True) — uses the HNSW index for the vector leg instead of the brute-force scan.

Below we index 8,000 code snippets with static_code_embedder (a fast model2vec static embedder) and compare the two vector legs.

import time

emb = static_code_embedder()                    # model2vec potion-code-16M — 256-dim float16
de, qe = doc_encoder(emb), query_encoder(emb)   # -> emb_fn: list[str] -> vectors

adb   = database()
store = adb.get_store('code', hash=True, ann=True)   # dtype defaults to float16, matching the embedder

# 8,000 code snippets; sync embeds, upserts, and builds the HNSW index in one call
docs = [dict(content=f'def func_{i}(x, y={i}): "helper {i%97}"; return x*{i} + y - {i%13}') for i in range(8000)]
t = time.perf_counter(); stats = store.sync(docs, emb_fn=de)
print(f'sync {stats} in {time.perf_counter()-t:.2f}s — index size {adb.get_index("code").size}')
# sync {'changed': 8000, 'same': 0, 'removed': 0} in 0.57s — index size 8000

sync {'changed': 8000, 'same': 0, 'removed': 0} in 0.63s — index size 8000

q  = 'function that multiplies its input by a constant and adds an offset'
qv = qe([q])[0].tobytes()

def bench(fn, reps=20):
    t = time.perf_counter()
    for _ in range(reps): fn()
    return (time.perf_counter()-t)/reps*1000   # ms per call

bf = bench(lambda: store.vec_search(qv, columns=['content'], limit=10))   # exact brute-force scan
an = bench(lambda: store.ann_search(qv, columns=['content'], limit=10))   # HNSW
print(f'brute-force {bf:.3f} ms   ann {an:.3f} ms   speedup {bf/an:.1f}x')
# brute-force 1.787 ms   ann 0.100 ms   speedup 17.8x

# HNSW is approximate but recall is high — top-10 matches the exact scan here
exact = {r['content'] for r in store.vec_search(qv, columns=['content'], limit=10)}
approx = {r['content'] for r in store.ann_search(qv, columns=['content'], limit=10)}
print(f'recall@10: {len(exact & approx)}/10')
# recall@10: 10/10

brute-force 1.889 ms   ann 0.105 ms   speedup 18.1x
recall@10: 9/10

# use it end-to-end via db.search — ann=True swaps in the HNSW index for the vector leg
adb.search(q, qv, columns=['content'], table_name='code', ann=True, limit=3)

[{'content': 'def func_3601(x, y=3601): "helper 12"; return x*3601 + y - 0',
  'rowid': 3602,
  '_dist': 0.7722985148429871,
  '_rrf_score': 0.016666666666666666},
 {'content': 'def func_3605(x, y=3605): "helper 16"; return x*3605 + y - 4',
  'rowid': 3606,
  '_dist': 0.7867892384529114,
  '_rrf_score': 0.01639344262295082},
 {'content': 'def func_3608(x, y=3608): "helper 19"; return x*3608 + y - 7',
  'rowid': 3609,
  '_dist': 0.7960468530654907,
  '_rrf_score': 0.016129032258064516}]

litesearch.data

Query Preprocessing

FTS5 is powerful, but raw natural-language queries often miss results. litesearch.data ships helpers to transform queries before sending them to FTS:

q = 'This is a sample query'
print('preprocessed q with defaults: `%s`' % pre(q))
print('keywords extracted: `%s`'          % pre(q, wc=False, wide=False))
print('q with wild card: `%s`'            % pre(q, extract_kw=False, wide=False, wc=True))

preprocessed q with defaults: `sample* OR query*`
keywords extracted: `sample query`
q with wild card: `This* is* a* sample* query*`

Function	What it does
clean(q)	strips * and returns None for empty queries
add_wc(q)	appends * to each word for prefix matching
mk_wider(q)	joins words with OR for broader matching
kw(q)	extracts keywords via YAKE (removes stop-words)
pre(q)	applies all of the above in one call

PDF Extraction

litesearch.data patches pdf_oxide.PdfDocument with bulk page-extraction methods. All methods take optional st / end page indices and return a fastcore L list:

Method	Returns
doc.pdf_texts(st, end)	plain text per page
doc.pdf_markdown(st, end)	markdown with headings + tables detected
doc.pdf_links(st, end)	URI strings extracted from annotations
doc.pdf_tables(st, end)	structured rows / cells / bbox dicts
doc.pdf_spans(st, end)	text spans with font size, weight, bbox
doc.pdf_images(st, end, output_dir)	image metadata, or save to disk
doc.pdf_chunks(st, end)	(page, chunk_idx, text) triples, markdown-chunked via chonkie

images_to_pdf(imgs, output) goes the other direction — wraps a list of images (PIL Images, bytes, or paths) into a conformant multi-page image-only PDF with no external dependencies.

doc = PdfDocument('pdfs/attention_is_all_you_need.pdf')

print(f'{doc.page_count()} pages, {len(doc.pdf_links())} links')
# plain text of page 1
doc.pdf_texts(0, 1)[0][:300]

Dictionary used where Stream expected, treating as empty stream
Dictionary used where Stream expected, treating as empty stream
Dictionary used where Stream expected, treating as empty stream
Dictionary used where Stream expected, treating as empty stream

15 pages, 18 links

'Provided proper attribution is provided, Google hereby grants permission to\nreproduce the tables and figures in this paper solely for use in journalistic or scholarly works.\n\n\nAttention\n\n\nAshish Vaswani∗ Noam Shazeer\n\n\n Is \n\n\n∗\n\n\nAll You\n\n\nNiki \n\n\n \n\n\nParmar\n\n\nNeed\n\n\n∗ Jakob Uszkoreit∗\n\n\nIllia Polos'

# markdown export — headings and tables are detected automatically
md = doc.pdf_markdown()
print(f'Page 1 (markdown):\n{md[0][:400]}')

Page 1 (markdown):
Provided proper attribution is provided, Google hereby grants permission to reproduce the tables and figures in this paper solely for use in journalistic or scholarly works.

## Attention Is All You Need

**Ashish** **Vaswani**

**Noam** **Shazeer**

**Niki** **Parmar**

**Jakob** **Uszkoreit** Google Brain

Google Brain

Google Research Google Research [[e-mail]](mailto:avaswani@google

doc.pdf_chunks() wraps pdf_markdown() + chonkie’s RecursiveChunker into (page, chunk_idx, text) triples — the direct input for encode_pdf_texts:

chunks = doc.pdf_chunks()
print(f'{len(chunks)} chunks from {doc.page_count()} pages')
# 31 chunks from 15 pages

# (page, chunk_idx, text) triples — direct input for encode_pdf_texts
pg, ci, text = chunks[0]
print(f'page {pg}, chunk {ci}: {text[:80]}...')

16 chunks from 15 pages
page 0, chunk 0: Provided proper attribution is provided, Google hereby grants permission to repr...

Code & File Ingestion

pyparse splits a Python file or string into top-level code chunks (functions, classes, assignments) with source location metadata — ready to insert into a store:

txt = """
from fastcore.all import *
a=1
class SomeClass:
    def __init__(self,x): store_attr()
    def method(self): return self.x + a
"""
pyparse(code=txt)

[{'content': 'class SomeClass:\n    def __init__(self,x): store_attr()\n    def method(self): return self.x + a', 'metadata': {'path': 'None', 'uploaded_at': None, 'name': 'SomeClass', 'lang': '.py', 'type': 'ClassDef', 'lineno': 4, 'end_lineno': 6}}]

pkg2chunks indexes an entire installed package in one call — great for building a semantic code-search store over your dependencies:

chunks = pkg2chunks('fastlite')
print(f'{len(chunks)} chunks from fastlite')
chunks.filter(lambda d: d['metadata']['type'] == 'FunctionDef')[0]

47 chunks from fastlite

{'content': 'def t(self:Database): return _TablesGetter(self)',
 'metadata': {'path': '/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/fastlite/core.py',
  'uploaded_at': 1773452878.5692947,
  'name': 't',
  'lang': '.py',
  'type': 'FunctionDef',
  'lineno': 44,
  'end_lineno': 44,
  'package': 'fastlite',
  'version': '0.2.4'}}

file_parse is the single entry point for any file type — Python, Jupyter notebooks, PDF, Markdown, plain text, and compiled-language source files (JS/TS, Go, Java, Rust…). All return the same {content, metadata} dicts:

# Python → AST-parsed functions and classes
py=file_parse(repo_root()/'litesearch/core.py')[:2]

# Jupyter notebook → one dict per cell
nb=file_parse(repo_root()/'nbs/01_core.ipynb')[:2]

# PDF → markdown-chunked text (via pdf_chunks)
pdf=file_parse(Path('pdfs/attention_is_all_you_need.pdf'))[:2]
print(py[0], '\n------------\n', nb[0], '\n---------------\n', pdf[0])

Dictionary used where Stream expected, treating as empty stream
Dictionary used where Stream expected, treating as empty stream
Dictionary used where Stream expected, treating as empty stream
Dictionary used where Stream expected, treating as empty stream

{'content': "def _dtype_suffix(dtype=np.float16): return _dtype_suffixes.get(dtype, 'f32')", 'metadata': {'path': '/Users/71293/code/litesearch/litesearch/core.py', 'uploaded_at': 1784347629.5078557, 'name': '_dtype_suffix', 'lang': '.py', 'type': 'FunctionDef', 'lineno': 15, 'end_lineno': 15}} 
------------
 {'content': '---\ndescription: Building blocks for litesearch\noutput-file: core.html\ntitle: core\n\n---\n\n', 'metadata': {'path': '/Users/71293/code/litesearch/nbs/01_core.ipynb', 'uploaded_at': 1784345054.1480775, 'lang': '.ipynb', 'type': 'raw'}} 
---------------
 {'content': 'Provided proper attribution is provided, Google hereby grants permission to reproduce the tables and figures in this paper solely for use in journalistic or scholarly works.\n\n## Attention Is All You Need\n\n**Ashish** **Vaswani**\n\n**Noam** **Shazeer**\n\n**Niki** **Parmar**\n\n**Jakob** **Uszkoreit** Google Brain\n\nGoogle Brain\n\nGoogle Research Google Research [[e-mail]](mailto:[e-mail]) [[e-mail]](mailto:[e-mail]) [[e-mail]](mailto:[e-mail]) [[e-mail]](mailto:[e-mail])\n\n**Llion** **Jones**\n\n**Aidan** **N.** **Gomez**∗ †\n\n**Łukasz** **Kaiser** Google Research\n\nUniversity of Toronto\n\nGoogle Brain [[e-mail]](mailto:[e-mail])\n\n[[e-mail]](mailto:[e-mail]) [[e-mail]](mailto:[e-mail])\n\n**Illia** **Polosukhin**∗ ‡ [[e-mail]](mailto:[e-mail])\n\n#### Abstract\n\nThe dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to\n\narXiv:1706.03762v7 [cs.CL] 2 Aug 2023  other tasks by applying it successfully to English constituency parsing both with large and limited training data.\n\nEqual contribution. Listing order is random. Jakob proposed replacing RNNs with self-attention and started the effort to evaluate this idea. Ashish, with Illia, designed and implemented the first Transformer models and has been crucially involved in every aspect of this work. Noam proposed scaled dot-product attention, multi-head attention and the parameter-free position representation and became the other person involved in nearly every detail. Niki designed, implemented, tuned and evaluated countless model variants in our original codebase and tensor2tensor. Llion also experimented with novel model variants, was responsible for our initial codebase, and efficient inference and visualizations. Lukasz and Aidan spent countless long days designing various parts of and implementing tensor2tensor, replacing our earlier codebase, greatly improving results and massively accelerating our research. Work performed while at Google Brain. Work performed while at Google Research.\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n', 'metadata': {'path': Path('pdfs/attention_is_all_you_need.pdf'), 'uploaded_at': 1773199216.9632287, 'lang': '.pdf'}}

dir2chunks indexes every file in a directory tree — analogous to pkg2chunks but for arbitrary directories rather than installed packages:

# Index all Python source files in a directory
chunks = dir2chunks(repo_root()/'litesearch', types='py')
print(f'{len(chunks)} chunks from litesearch/')

# Mix formats: notebooks, markdown, PDFs
chunks = dir2chunks(repo_root()/'nbs', types='ipynb,md,pdf')
print(f'{len(chunks)} chunks from nbs/')

Dictionary used where Stream expected, treating as empty stream
Dictionary used where Stream expected, treating as empty stream
Dictionary used where Stream expected, treating as empty stream
Dictionary used where Stream expected, treating as empty stream

63 chunks from litesearch/
2910 chunks from nbs/

litesearch.utils

FastEncode — ONNX Text Encoder

FastEncode wraps any ONNX model from HuggingFace Hub. It handles tokenisation, batching, optional parallel thread-pool execution, and runtime int8 quantization — all without PyTorch or Transformers.

Config	Model	Dim	Notes
embedding_gemma (default)	onnx-community/embeddinggemma-300m-ONNX	768	Strong retrieval, ~300M params
modernbert	nomic-ai/modernbert-embed-base	768	BERT-style, fast
nomic_text_v15	nomic-ai/nomic-embed-text-v1.5	768	Shares embedding space with nomic_vision_v15

encode_document and encode_query apply the model’s prompt templates automatically.

texts = [
    'Attention is all you need',
    'The transformer architecture uses self-attention',
    'BERT pretrains on masked language modeling',
    'GPT uses autoregressive generation',
]

# Default model — downloads once, cached
enc, enc_fast, enc_q = FastEncode(),FastEncode(batch_size=2, parallel=2),FastEncode(quantize='int8')

doc_embs=enc.encode_document(texts); doc_embs

array([[ 0.06824 ,  0.015434,  0.005802, ..., -0.04044 ,  0.02196 ,
         0.03586 ],
       [ 0.02335 , -0.0157  ,  0.0168  , ..., -0.08057 , -0.03833 ,
         0.0258  ],
       [ 0.03894 ,  0.01215 ,  0.00667 , ..., -0.06915 , -0.00802 ,
         0.0077  ],
       [ 0.02185 ,  0.01064 , -0.01266 , ..., -0.0686  , -0.0412  ,
         0.02237 ]], shape=(4, 768), dtype=float16)

q_emb = enc.encode_query(['what paper introduced transformers?'])
print('doc shape:', doc_embs.shape, 'dtype:', doc_embs.dtype)  # (4, 768) float16

doc shape: (4, 768) dtype: float16

# Batching + parallel thread-pool
embs = enc_fast.encode_document(texts); embs

<style> progress { appearance: none; border: none; border-radius: 4px; width: 300px; height: 20px; vertical-align: middle; background: #e0e0e0; } progress::-webkit-progress-bar { background: #e0e0e0; border-radius: 4px; } progress::-webkit-progress-value { background: #2196F3; border-radius: 4px; } progress::-moz-progress-bar { background: #2196F3; border-radius: 4px; } progress:not([value]) { background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px); } progress.progress-bar-interrupted::-webkit-progress-value { background: #F44336; } progress.progress-bar-interrupted::-moz-progress-value { background: #F44336; } progress.progress-bar-interrupted::-webkit-progress-bar { background: #F44336; } progress.progress-bar-interrupted::-moz-progress-bar { background: #F44336; } progress.progress-bar-interrupted { background: #F44336; } table.fastprogress { border-collapse: collapse; margin: 1em 0; font-size: 0.9em; } table.fastprogress th, table.fastprogress td { padding: 8px 12px; border: 1px solid #ddd; text-align: left; } table.fastprogress thead tr { background: #f8f9fa; font-weight: bold; } table.fastprogress tbody tr:nth-of-type(even) { background: #f8f9fa; } </style>

array([[ 0.06824 ,  0.015434,  0.005802, ..., -0.04044 ,  0.02196 ,
         0.03586 ],
       [ 0.02335 , -0.0157  ,  0.0168  , ..., -0.08057 , -0.03833 ,
         0.0258  ],
       [ 0.03894 ,  0.01215 ,  0.00667 , ..., -0.06915 , -0.00802 ,
         0.0077  ],
       [ 0.02185 ,  0.01064 , -0.01266 , ..., -0.0686  , -0.0412  ,
         0.02237 ]], shape=(4, 768), dtype=float16)

# Runtime int8 quantization — creates model_int8.onnx on first run, reused after
enc_q.encode_document(texts)

array([[ 0.09485 ,  0.01581 ,  0.005184, ..., -0.05066 ,  0.01775 ,
         0.02722 ],
       [ 0.02164 , -0.01174 ,  0.0182  , ..., -0.0741  , -0.03018 ,
         0.02805 ],
       [ 0.03583 ,  0.012405,  0.005947, ..., -0.05457 , -0.00577 ,
         0.003704],
       [ 0.02808 ,  0.02591 , -0.009514, ..., -0.05954 , -0.04718 ,
         0.01416 ]], shape=(4, 768), dtype=float16)

FastEncodeImage — ONNX Image Encoder

FastEncodeImage encodes images with CLIP-style ONNX vision models. No Transformers dependency — preprocessing (resize → normalise → CHW) is done with PIL + NumPy using config stored in the model dict.

Config	Model	Dim	Notes
nomic_vision_v15 (default)	nomic-ai/nomic-embed-vision-v1.5	768	Same space as nomic_text_v15
clip_vit_b32	Qdrant/clip-ViT-B-32-vision	512	Classic CLIP

Accepts PIL Images, file paths, or raw bytes — any mix.

FastEncodeMultimodal — Cross-Modal Image + Text Search

FastEncodeMultimodal wraps a model repo that ships both text and vision ONNX encoders in a single shared embedding space — a text query can retrieve images directly. Below: index Attention Is All You Need (text chunks + figures) then search for 'attention mechanism diagram'.

Unified model — siglip2_so400m (~800 MB, one download):

import json, base64, io
from PIL import Image
from IPython.display import display

enc = FastEncodeMultimodal(siglip2_so400m)   # single unified model, ~800 MB, cached on first run
doc = PdfDocument('pdfs/attention_is_all_you_need.pdf')
db  = database()
ts, ims = db.get_store('texts'), db.get_store('images')

for pg, ci, chunk, emb in encode_pdf_texts(doc, enc.text):
    ts.insert(dict(content=chunk, embedding=emb.tobytes(), metadata=json.dumps({'page': pg})))
for pg, img_bytes, emb in encode_pdf_images(doc, enc.vision):
    ims.insert(dict(content=f'page_{pg}', embedding=emb.tobytes(),
                    metadata=json.dumps({'page': pg, 'data': base64.b64encode(img_bytes).decode()})))

q = 'attention mechanism diagram'
q_emb = enc.text.encode([q])[0].tobytes()
txt_r = ts.db.search(pre(q), q_emb, table_name='texts', columns=['content']) or []
img_r = ims.vec_search(q_emb)
for r in rrf_merge(txt_r, img_r)[:6]:
    print(f"rrf={r['_rrf_score']:.4f}  {r['content'][:70]}")
    meta = json.loads(r.get('metadata', '{}'))
    if 'data' in meta:
        display(Image.open(io.BytesIO(base64.b64decode(meta['data']))).resize((200, 150)))

/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
Fetching 36 files:  14%|█▍        | 5/36 [04:04<34:59, 67.74s/it]

Paired models — nomic_text_v15 + nomic_vision_v15 share the same 768-dim space; use FastEncode and FastEncodeImage separately:

enc_text = FastEncode(nomic_text_v15)
enc_img  = FastEncodeImage(nomic_vision_v15)
db2  = database()
ts2, ims2 = db2.get_store('texts'), db2.get_store('images')

for pg, ci, chunk, emb in encode_pdf_texts(doc, enc_text):
    ts2.insert(dict(content=chunk, embedding=emb.tobytes(), metadata=json.dumps({'page': pg})))
for pg, img_bytes, emb in encode_pdf_images(doc, enc_img):
    ims2.insert(dict(content=f'page_{pg}', embedding=emb.tobytes(),
                     metadata=json.dumps({'page': pg, 'data': base64.b64encode(img_bytes).decode()})))

q_emb2 = enc_text.encode([q])[0].tobytes()
txt_r2 = ts2.db.search(pre(q), q_emb2, table_name='texts', columns=['content']) or []
img_r2 = ims2.vec_search(q_emb2)
for r in rrf_merge(txt_r2, img_r2)[:6]:
    print(f"rrf={r['_rrf_score']:.4f}  {r['content'][:70]}")
    meta = json.loads(r.get('metadata', '{}'))
    if 'data' in meta:
        display(Image.open(io.BytesIO(base64.b64decode(meta['data']))).resize((200, 150)))

rrf=0.0167  Self-attention, sometimes called intra-attention is an attention mecha
rrf=0.0167  page_3

rrf=0.0164  Attention mechanisms have become an integral part of compelling sequen
rrf=0.0164  page_2

rrf=0.0161  2,[19]. Inall but a few cases27],[ however, such attention mechanisms
rrf=0.0161  page_3

rrf=0.0167  Self-attention, sometimes called intra-attention is an attention mecha
rrf=0.0167  page_3

rrf=0.0164  Attention mechanisms have become an integral part of compelling sequen
rrf=0.0164  page_2

rrf=0.0161  2,[19]. Inall but a few cases27],[ however, such attention mechanisms
rrf=0.0161  page_3

Next Steps

• examples/01_simple_rag.ipynb — ingest a folder of PDFs, chunk with chonkie, rerank with FlashRank
• examples/02_tool_use.ipynb — wire litesearch into an LLM tool-use loop
• core docs — full API reference for database, get_store, search, rrf_merge, vec_search
• data docs — PDF methods, pyparse, pkg2chunks, query preprocessing
• utils docs — FastEncode, download_model, image tools

Acknowledgements

A big thank you to @yfedoseev for pdf-oxide, which powers the PDF extraction functionality in litesearch.data.