bilbysearch

bilbysearch turns articles into vectors with intfloat/multilingual-e5-large-instruct, and reads and writes the daily parquet files those vectors live in.

It exists so the model's contract lives in one place. The model id, the 1024-d output width, the instruction prefix queries need and the title+body join rule are all in config.py, so notebooks, batch backfills and retrieval pipelines cannot drift apart.

Install

pip install bilbysearch[embed]

The base install is deliberately light — just pandas and pyarrow — so a caller that only reads the parquet files, or only reads the model contract, does not pull a multi-gigabyte ML stack it will never use.

Until the first PyPI release, install from the repo instead:

uv add "bilbysearch[all] @ git+https://github.com/bilbyai/bilbysearch.git"

Usage

import bilbysearch

model = bilbysearch.load_embedder()

# Embed a frame of articles. Rows with no text get a null vector rather than a
# meaningless one, and three provenance columns record how the vectors were made.
articles = bilbysearch.read_parquet("daily_2026-01-31.parquet")
embedded = bilbysearch.embed_dataframe(articles, model=model)
bilbysearch.write_parquet(embedded, "daily_2026-01-31_embedding.parquet")

Queries and passages are embedded by different functions on purpose: this model expects an instruction prefix on queries and none on passages. Skipping it measurably degrades retrieval.

query = bilbysearch.embed_query("Chinese monetary policy easing", model=model)
passages = bilbysearch.embed_texts(["The PBoC cut the reserve requirement ratio."], model=model)
similarity = query @ passages[0]  # vectors are L2-normalised, so this is cosine

Only load_embedder needs the [embed] extra. Every other function works on any object exposing sentence-transformers' encode, so a caller that already has a model can pass it straight in.

Development

uv sync
uv run pytest
uv run ruff check . && uv run ruff format .

The tests use a stub model, so the suite runs in well under a second and never downloads the real 2.2 GB one.

Licence

MIT — see LICENSE.