bilbysearch

This repo is a work in progress - it is a package for dedicated search code in future search pipelines at Bilby.

bilbysearch turns articles into vectors - by default with intfloat/multilingual-e5-large-instruct, or with any of the 31 other models in its registry, locally or through OpenRouter - and reads and writes the daily parquet files those vectors live in. It searches those articles by meaning (vector search) or by keyword (BM25, in English or Chinese), finds which names they mention by the aliases each name goes by, combines a search by meaning with those mentions into one list of candidates, and reranks the results - with a cross-encoder on this machine, a reranker on OpenRouter, or Jev. Long articles can be embedded and searched a chunk at a time, so text past a model's context window is not lost, and a search's ranking can be scored against relevance labels - recall, precision, nDCG and AP. It also reads Postgres databases into DataFrames, read-only: the entities database - the people entity_profiles tracks - or any other.

It exists so each model's contract lives in one place. What a model expects - its output width, context window, query instruction and document prefix - is recorded once, in embedding_model_config.py, and every function takes the model by name and applies that record. Which model production uses, and pipeline rules such as the title+body join, are in production_config.py. Rerankers have a registry of their own, reranker_config.py. So notebooks, batch backfills and retrieval pipelines cannot drift apart.

Install

pip install bilbysearch

That one command brings embedding, storage, search and reranking: the model stack (sentence-transformers, torch), gs:// support (fsspec, gcsfs, google-cloud-storage), BM25 keyword search (bm25s, and jieba to split Chinese into words), alias search (pyahocorasick), reranking with a local cross-encoder or through OpenRouter, read-only Postgres queries (psycopg, whose wheel bundles the Postgres client library), and the retrieval metrics (scikit-learn, which sentence-transformers brings anyway). torch is the heavy part, at around 2 GB once CUDA wheels are involved.

Only reranking with Jev is optional, because it brings bilbystats and its LLM client libraries. The extra is called rerank because it predates the other rerankers, which need nothing extra:

pip install "bilbysearch[rerank]"

[all] installs everything. The [embed] and [gcs] extras from 0.1.x are still accepted, so an existing "bilbysearch[embed,gcs]" dependency keeps working, but they no longer add anything.

Tutorials

tutorials/ walks through the package in notebooks, each runnable top to bottom on a few articles built in the notebook. Start with getting_started.ipynb, which embeds a day of articles, stores it, searches it by meaning and by keyword, and reranks what the searches find. The index lists the rest in reading order, with what each needs to run: most need nothing set up, and those that call OpenRouter say how many requests they make. Each section below links to the tutorial for its part.

Usage

import bilbysearch

# Embed a frame of articles. Rows with no text get a null vector rather than a
# meaningless one, and three provenance columns record how the vectors were made.
articles = bilbysearch.read_parquet("daily_2026-01-31.parquet")
embedded = bilbysearch.embed_dataframe(articles)
bilbysearch.write_parquet(embedded, "daily_2026-01-31_embedding.parquet")

Queries and documents are embedded by different functions on purpose: the production model expects an instruction prefix on queries and none on documents, and skipping it measurably degrades retrieval. embed_query adds the query instruction and embed_texts the document prefix, so neither can be forgotten or put on the wrong side.

query = bilbysearch.embed_query("Chinese monetary policy easing")
passages = bilbysearch.embed_texts(["The PBoC cut the reserve requirement ratio."])
similarity = query @ passages[0]  # vectors are L2-normalised, so this is cosine

The model is loaded on first use and kept loaded - the two most recently used models stay in memory - so embedding a frame and then a query loads it once. clear_embedder_cache() releases them. Every function also takes device ("cuda", "mps" or "cpu"; the default lets sentence-transformers choose).

Progress - which model is loading, how many rows are embedded, each OpenRouter batch - is logged at INFO, through loggers named after the modules (bilbysearch.embedding.embed, bilbysearch.storage, ...). The package configures no logging of its own: warnings, such as an OpenRouter request being retried, reach stderr with no setup, and to see the progress too, ask for it:

import logging

logging.basicConfig()
logging.getLogger("bilbysearch").setLevel(logging.INFO)

Choosing a model

Every function takes the model's name as model_name, defaulting to the production model. The name is all you pass: its width, context window and prefixes come from its entry in the registry.

bilbysearch.list_local_embedding_models(language="zh")  # "en", "zh" or "multilingual"
bilbysearch.get_model_spec("BAAI/bge-m3")                # what the registry records

embedded = bilbysearch.embed_dataframe(articles, model_name="intfloat/e5-large-v2")
query = bilbysearch.embed_query("PBoC easing", "intfloat/e5-large-v2")  # gets "query: "

That dataframe's articles were embedded with e5-large-v2's "passage: " prefix, and its embedding_model column says intfloat/e5-large-v2: the one name picks the model and labels the output, so the two cannot disagree. Embed queries with the same model as the documents; vectors from two models are not comparable.

Mistakes fail before anything is loaded or sent:

• An unknown name is refused, with the nearest known names suggested ("baai/bge-m3" -> did you mean 'BAAI/bge-m3'?).
• A text that already starts with its model's prefix is refused, e.g. "passage: ..." for e5-large-v2, which would otherwise be embedded as "passage: passage: ...". Pass texts without prefixes.
• A loaded model that disagrees with its registry width is refused, so its vectors never reach a vector store sized from the registry.

To switch production to another model, change MODEL_NAME in production_config.py to a local model from the registry; its width, context window and prefixes follow. tests/embedding/test_production_config.py pins the production values, so update it in the same change. Existing vectors must be re-embedded, and a vector store of a different width recreated.

To add an embedding model, add its entry to embedding_model_config.py: its output width (measured from real output, not read from its card), context window, query instruction and document prefix (copied character for character from its card), language, whether it needs trust_remote_code, and where it runs.

local_small_model_embeddings.ipynb runs every embedding step with a 90 MB model, and shows how to try another.

Through OpenRouter

The same steps can run on OpenRouter's servers instead of this machine, for the models the registry records as hosted there (list_openrouter_embedding_models(); the production model is not one of them). Set OPENROUTER_API_KEY, then pass the model's registry name - the id OpenRouter uses and the model's prefixes come from its entry:

model_name = "intfloat/multilingual-e5-large"

embedded = bilbysearch.embed_dataframe_openrouter(articles, model_name=model_name)
passages = bilbysearch.embed_texts_openrouter(["The PBoC cut the RRR."], model_name)
query = bilbysearch.embed_query_openrouter("PBoC easing", model_name)

Locally, a text longer than the model's context window is cut short: e5-large reads only its first 512 tokens. OpenRouter would receive the whole text, so where the registry names the model's tokenizer (e5-large and the Perplexity models), each text is cut to the same window with that tokenizer before it is sent, and OpenRouter embeds exactly what the local model reads. OpenAI's models refuse a text over their 8,191-token window outright - one such text fails its whole batch - so their texts are cut with tiktoken, OpenAI's own tokenizer, before they are sent. Gemini's tokenizer is not public, so its texts are sent whole, and OpenRouter cuts each to Gemini's 2,048-token window itself, keeping the start. It bills the whole text, though, so a long article costs more than the part that is embedded.

Every text is also capped at 256 KiB (MAX_INPUT_BYTES in embed_openrouter.py), between whole characters. A text that is mostly whitespace can fit inside a model's window and still be megabytes long, and the provider's web server refuses such a request (HTTP 413) before any model reads it. A genuine article cut to its window stays well under the cap.

To count the tokens OpenRouter bills, pass a dict as usage. Each response's counts - prompt_tokens, total_tokens, and any other count OpenRouter reports - are added to it as the response arrives, so one dict totals a whole run, and a run that fails partway still counts the batches already answered:

usage = {}
embedded = bilbysearch.embed_dataframe_openrouter(articles, model_name=model_name, usage=usage)
query = bilbysearch.embed_query_openrouter("PBoC easing", model_name, usage=usage)
usage["prompt_tokens"]  # every token billed for the frame and the query

Before mixing OpenRouter and local vectors in one index, check they match:

report = bilbysearch.compare_embeddings(local_vectors, openrouter_vectors)
report["all_match"]  # True if every pair has cosine similarity >= 0.99

check_openrouter_matches_local.py runs this check for multilingual-e5-large.

Threading

embed_texts_openrouter and embed_dataframe_openrouter send texts in batches of batch_size (default 32) per request, and use threads to keep up to max_concurrent_requests (default 8) requests in flight at once. Vectors come back in input order. Lower max_concurrent_requests if OpenRouter rate-limits you; 1 sends one request at a time.

embedded = bilbysearch.embed_dataframe_openrouter(
    articles, model_name=model_name, max_concurrent_requests=4
)

For async code, embed_texts_openrouter_async and embed_query_openrouter_async take the same arguments and can be awaited without blocking the event loop:

query = await bilbysearch.embed_query_openrouter_async("PBoC easing", model_name)

embed_openrouter.ipynb runs each of these, and checks OpenRouter's vectors against the same model run locally.

Upgrading from 0.8

0.9 groups the modules by job - embedding/, vector_search/ and keyword_search/, beside reranking/ and entities/ - so a module's path says which part of the pipeline it belongs to. Everything imported from bilbysearch itself, such as from bilbysearch import embed_dataframe, is unchanged. A moved module imported by its old path fails at the import, with ModuleNotFoundError:

0.8	0.9
bilbysearch.embed	bilbysearch.embedding.embed
bilbysearch.embed_openrouter	bilbysearch.embedding.embed_openrouter
bilbysearch.embedding_model_config	bilbysearch.embedding.embedding_model_config
bilbysearch.production_config	bilbysearch.embedding.production_config
bilbysearch.chunking	bilbysearch.embedding.chunking
bilbysearch.compare	bilbysearch.embedding.compare
bilbysearch.bm25_search	bilbysearch.keyword_search.bm25_search
bilbysearch.vector_search	a package at the same path: vector_search, CHUNK_SCORINGS and DEFAULT_TOP_K import from it as before, and the module's other names, such as stack_chunk_vectors, are in bilbysearch.vector_search.vector_search
OPENROUTER_API_KEY_ENV, RETRYABLE_STATUSES or openrouter_error from bilbysearch.embed_openrouter	from bilbysearch.openrouter_client, or OPENROUTER_API_KEY_ENV from bilbysearch
LANGUAGES from bilbysearch.embedding_model_config	from bilbysearch
bilbysearch.model_config, deprecated since 0.7	removed: bilbysearch.embedding.embedding_model_config, or from bilbysearch import MODELS

Upgrading from 0.6

0.7 adds local and OpenRouter rerankers, and with them a reranker registry, so the embedding registry's module is renamed to say which models it holds. Code written for 0.6 keeps running, unless it passed rerank_jev an argument that could never be right:

0.6	0.7
from bilbysearch.model_config import MODELS	from bilbysearch.embedding_model_config import MODELS, or from bilbysearch import MODELS. The old import kept working, with a DeprecationWarning, until 0.9 removed it.
rerank_jev(q, df, top_n=-1) quietly kept all but the last row	raises ValueError, as do a top_n or max_chars below 1 and a min_relevance outside 0 to 1 - before any request is made

Upgrading from 0.2

0.3 takes models by name. Code written for 0.2 fails with an error saying what to pass instead, rather than running with the wrong settings:

0.2	0.3
model = load_embedder(name), then embed_texts(texts, model)	embed_texts(texts, name)
embed_query(q, model, instruction="query: ")	embed_query(q, name)
embed_dataframe(df, model=model, model_name=name)	embed_dataframe(df, model_name=name)
embed_texts(["passage: " + t for t in texts], model)	embed_texts(texts, name)
embed_query_openrouter(q, name, instruction="query: ")	embed_query_openrouter(q, name)
vector_search(q, df, model, instruction=...)	vector_search(q, df, name)
from bilbysearch.config import ...	from bilbysearch.production_config import ..., or from bilbysearch import ...
any model id	a name in bilbysearch.MODELS

Searching

vector_search scores every article in an embedded frame against a query and returns the closest, best first, with a cosine_similarity column. To search stored days, read them into one frame first:

uris = [bilbysearch.parquet_uri(prefix, date, "_embedding") for date in dates]
embedded = pd.concat([bilbysearch.read_parquet(uri) for uri in uris], ignore_index=True)
candidates = bilbysearch.vector_search("Chinese monetary policy easing", embedded, top_k=100)

The query is embedded with model_name - the production model unless you say otherwise - and it has to be the model that made the article vectors. Frames from embed_dataframe record that model in their embedding_model column, so a search with a different one, or over a frame that mixes days embedded by two models, is refused rather than returning a meaningless ranking.

It is brute force: every vector is loaded and compared in memory. That is exact and fine for weeks of articles; a corpus too big to load at once needs a vector store.

A subject that spans several topics is better searched with a query per topic, since one query vector for all of them matches none of them well. multi_vector_search takes the queries, gives each its own top_k, and returns an article that more than one query finds once:

queries = ["NEV sales and price wars", "battery technology", "exports and overseas plants"]
candidates = bilbysearch.multi_vector_search(queries, embedded, top_k=30)

An article's cosine_similarity is its best over every query, and best_query is the index of the query that gave it. matched_queries lists the queries whose top_k holds it, which need not include best_query: an article can score highest on a query whose top_k is full of closer ones. The other arguments are vector_search's.

vector_search.ipynb searches embedded days, and storage.ipynb covers the daily files they are read from, and a backfill that embeds each day once.

Keyword search with BM25

vector_search matches meanings; bm25_search matches words. It finds exact names, codes and figures that an embedding can blur, and misses the paraphrases an embedding catches. It needs no embeddings, so it runs on the raw daily files. Build an index once, then search it as often as needed:

day = bilbysearch.read_parquet("daily_2026-01-31.parquet")
index = bilbysearch.build_bm25_index(day, language="zh")  # title+body, split with jieba
hits = bilbysearch.bm25_search("央行 降准", index)         # milliseconds per search
hits[["title", "bm25_score"]]

Choose the language and the columns together. A query only matches words in its own language, and each language is split into words differently. In the daily files title+body is mostly Chinese and title_en+body_en is English:

en_index = bilbysearch.build_bm25_index(day, language="en", text_col="title_en+body_en")
hits = bilbysearch.bm25_search("reserve requirement cut", en_index)

• "en" splits text the way bm25s does by default - lowercased words of two or more letters or digits, English stopwords removed - so results match the team's English BM25 baseline.
• "zh" splits Chinese into words with jieba's search mode, which also emits the shorter words inside long ones: 中国人民银行 gives 中国, 人民, 银行 and the whole name, so a query for any of them finds it. Full-width letters and digits (ＧＤＰ, ４.８％) read as GDP and 4.8%, and Latin terms in Chinese text (PBoC, 5G) are kept.

language has no default, and an English index refuses a query with Chinese in it rather than silently matching nothing.

What comes back. Up to top_k (default 100) articles that contain at least one of the query's words, best first, with a bm25_score column - ready to rerank. Articles with none of the words are left out, so fewer than top_k can come back. A score depends on the whole index as well as the article - how rare each word is across it, how long its articles are on average - so it ranks articles within one search. It is not comparable across indexes, and is not a relevance cutoff.

Tuning. build_bm25_index takes BM25's two parameters. k1 (default 1.5) sets how much repeating a word raises a score: higher lets each repeat keep adding, lower makes a word count about the same however often it appears. b (default 0.75) sets how much long articles are discounted, from 0 (not at all) to 1 (in proportion to their length). bm25s works out each word's contribution to each article's score when the index is built, which is what makes searches fast, so k1 and b are fixed for an index: build another to try other values.

Cost. Building reads every article: 2-3 s on a laptop for a day's Chinese title+body (around 1,300 articles), and 0.3 s for its English translations. Each search after that takes milliseconds. The index lives in memory.

Known limits of "zh":

• jieba's dictionary is simplified Chinese, so traditional text is split poorly.
• A name the dictionary does not know is split into pieces - 宁德时代 gives 宁德 and 时代 - which match more loosely than the whole name would.
• Search mode trades some precision for recall: 人民币 (the yuan) also gives 人民, which is in 中国人民银行 (the People's Bank of China), so a query about the yuan also finds People's Bank articles, lower down.

ex_bm25_search.py runs these searches on a few articles, and bm25_search.ipynb shows what each language is split into, tunes k1 and b, and sets BM25 beside vector_search on a stock code and a paraphrase.

Finding names by their aliases

bm25_search ranks the articles that match a query; find_aliases finds which names each article mentions, and where. Give it each name with the aliases it goes by. It reads a text once however many aliases there are - the Aho-Corasick algorithm - and returns every mention with the name it stands for and its offsets in the text:

index = bilbysearch.build_alias_index(
    {
        "Xi Jinping": ["Xi Jinping", "President Xi", "Xi", "习近平"],
        "United States": ["United States", "US", "美国"],
    },
    excluded_before=bilbysearch.TITLE_MODIFIERS,  # "Vice President X" is not "President X"
)
bilbysearch.find_aliases("President Xi told the US", index)
# [{'name': 'Xi Jinping', 'alias': 'President Xi', 'start': 0, 'end': 12},
#  {'name': 'United States', 'alias': 'US', 'start': 22, 'end': 24}]

# One row per mention - uuid, name, alias, start, end, and the field (column) it is in -
# to count or collect by article.
mentions = bilbysearch.find_aliases_dataframe(day, index, id_col="uuid", text_col="body_en")
mentions.groupby(["uuid", "name"]).size()

• Whole words. "Oman" does not match in "woman", nor "O" in "O'Brien", while a possessive 's still matches. Chinese has no spaces, so a Chinese alias needs no boundary around it: 习近平 matches in 习近平主席.
• The longest alias wins where aliases overlap, whichever names they belong to: "Oman LNG" is no mention of Oman, and "Xi Jinping" is one mention, not a second through "Xi". An alias several names share, such as a surname, mentions each of them.
• Acronyms keep their case. An alias that is one word in capitals - US, G7, OPEC+ - matches only as written, so "us" is no mention of the United States. Every other alias ignores case; list an acronym's other spellings, such as Covid-19, as aliases too.
• Look-alike characters meet. Curly apostrophes, non-breaking hyphens and spaces, and full-width brackets fold to their ordinary forms in aliases and text alike, and the offsets still index the text as given.
• Decoys - aliases listed under the name None - take their span and mention nothing: with None: ["TAQA Arabia"], "TAQA Arabia" is no mention of TAQA.

start and end index the text that was searched: for a "+"-joined text_col, the columns joined as select_text_series joins them. field names the column of text_col each mention is in, such as title or body, so mentions in the title can be picked out without working out where it ends. An empty alias, or one with spaces at either end, is refused rather than skipped. An index of 60,000 aliases builds in under a tenth of a second, and a day of 1,300 English articles searches in about a tenth of a second.

Alias search is built on build_term_index and find_terms, in aho_corasick_search.py, which find a plain list of terms by the same rules, with no names; a term keeps its case only if it is listed in case_sensitive_terms. aho_corasick_search.ipynb demonstrates them, and alias_search.ipynb walks through alias search on a day of articles, with its limits. examples/keyword_search/alias_search/ has an example script per function.

Combining vector search and alias mentions

A query per topic finds the articles on a company's topics, and misses news about the company itself - its results, its people - which names it more than it discusses its industry. Alias mentions find that news, and miss the articles on its topics that never name it. hybrid_search takes both - the queries, and the rows find_aliases_dataframe returns - and returns each article once, in one list of candidates for a reranker:

# One index for every company followed, and the day's mentions found once.
index = bilbysearch.build_alias_index({"BYD": ["BYD", "比亚迪"], "CATL": ["CATL", "宁德时代"]})
mentions = bilbysearch.find_aliases_dataframe(embedded, index, id_col="uuid")

candidates = bilbysearch.hybrid_search(
    embedded,
    id_col="uuid",
    queries=["NEV sales and price wars", "battery technology", "exports and overseas plants"],
    mentions=mentions[mentions["name"] == "BYD"],
    top_k=30,  # per query
    keyword_top_k=50,  # mentioned articles kept, title mentions first
)
reranked = bilbysearch.rerank_jev("BYD", candidates)

• Which searches found each article. sources lists vector:<i> for each query whose top_k holds it and keyword if the keyword side kept it, and n_sources counts them.
• Each side's columns. multi_vector_search's - cosine_similarity, best_query, matched_queries - are null where it did not find an article. mention_count and in_title come from the mention rows for every article, even one keyword_top_k cut from the keyword side, which only sources records.
• Sorted for a reranker, by n_sources, then in_title, then cosine_similarity (nulls last), then mention_count, which rerankers keep for ties. n_sources counts each query, so an industry story every topic query finds can outrank the company's own news until a reranker reads them.
• The same story from two outlets - rows sharing a stripped title - is cut to the first; untitled rows are never merged. Pass dedupe_titles=False to keep every copy.
• Title mentions need the title searched. in_title is true when a mention's field is title_col, so find the mentions in a text_col that includes it, as the default title+body does.
• Either side alone works; with mentions only, no model is loaded and the frame needs no embeddings.

id_col has no default: every row needs an id, and no two the same. Mention rows from another frame or made without field, and a frame that already holds a column hybrid_search adds, are refused before anything is embedded. hybrid_search.ipynb walks through it on a day of articles, and ex_hybrid_search.py has examples.

Chunking long articles

A model reads at most its context window - 512 tokens for the production model - and silently drops the rest, so an article's one vector describes only its opening. In the English 50-query experiment in semantic-search-tests, 63% of articles were longer than that, and a passage further in cannot be found by its meaning, however well it matches.

With chunking=True, each article is split into overlapping chunks that the model reads whole, and every chunk is embedded. The frame keeps one row per article:

embedded = bilbysearch.embed_dataframe(articles, chunking=True)
embedded["chunk_embeddings"]  # a list of vectors per article, one per chunk
embedded["chunk_spans"]       # each chunk's [start, end] in the article's title+body

hits = bilbysearch.vector_search("semiconductor export controls", embedded, chunking=True)
hits[["title", "cosine_similarity", "best_chunk_span"]]

vector_search scores every chunk and each article by its best one, so an article comes back once however many of its chunks match, top_k counts articles, and best_chunk_span says which passage matched. The hits go to a reranker as they are.

To score each article by the average of its chunks instead, pass chunk_scoring="mean":

hits = bilbysearch.vector_search(query, embedded, chunking=True, chunk_scoring="mean")

The mean favours articles about the query throughout over one that matches in a single passage. In ex_vector_search.py, the sports round-up found by its last paragraph drops from second to third. "max", the default (DEFAULT_CHUNK_SCORING), is what semantic-search-tests benchmarked; "mean" was not tested there.

How articles are cut. A chunk is a window of the model's own tokens. chunk_size defaults to as many as fit beside the model's special tokens and document prefix - 510 for the production model - and chunk_overlap to an eighth of that, 64, so text near a cut is read with context on both sides:

embedded = bilbysearch.embed_dataframe(articles, chunking=True, chunk_size=256, chunk_overlap=32)

• A chunk is the article's own characters, never tokens decoded back into text, which can differ from the original (some tokenizers put spaces between Chinese characters).
• Every chunk is tokenized again, prefix and all, and shortened if it would run over. That check is needed: a chunk cut from the middle of a text can take a token or two more on its own than it did in place, and in a sample of mixed Chinese and English articles, a third of chunks cut at exactly 510 tokens ran over 512.
• An article that fits is one chunk, and gets exactly the vector it gets unchunked.
• chunk_size and chunk_overlap are refused without chunking=True, rather than ignored, and a chunk_size larger than the model reads is refused.
• A long-context model's default chunk is so large that most articles are a single chunk - bge-m3 reads 8,192 tokens, Qwen3 32,768. Pass a chunk_size to chunk them.

Storing and searching. Pass the same chunking everywhere the frame goes:

stored = bilbysearch.select_output_columns(embedded, id_col="uuid", chunking=True)
bilbysearch.write_parquet(stored, "daily_2026-01-31_embedding.parquet")

The stored file holds the id, the chunk vectors, their spans and the provenance - not the chunks' text, which stays in the daily file: bilbysearch.select_text_series(day)[row][start:end] is a chunk. A frame stored or searched with the wrong chunking is refused, saying which kind of vectors it holds. DEFAULT_CHUNKING in production_config.py is the default for all four functions, so one change switches embedding, storing and searching together - and means re-embedding every stored day.

Through OpenRouter, embed_dataframe_openrouter takes the same arguments. Chunks are cut with the HuggingFace tokenizer the registry names, so only e5-large and the Perplexity models can be chunked there; chunking OpenAI's models, which tiktoken cuts instead, or Gemini, whose tokenizer is not public, is refused. e5-large is cut into the same chunks locally and on OpenRouter.

When it helps. Chunking is off in production because, for the production model, it found no more: over 848 queries in each language, recall@100 moved by less than 0.001 (English 0.9918 to 0.9921, Chinese 0.9952 to 0.9947; the Chinese difference is not significant), while embedding took twice as long (112 minutes against 51 for the Chinese run). It pays off where the passage that matters sits past the model's window - ex_vector_search.py finds a sports round-up by its last paragraph only when chunked - and for weaker Chinese models: on 50 Chinese queries, recall@100 rose from 0.80 to 0.88 for zpoint and from 0.88 to 0.90 for text2vec, though it fell for multilingual-e5-large, from 0.95 to 0.92.

chunking.ipynb walks through all of this, on an English and a Chinese article whose most relevant passage the model would otherwise never read.

Reranking

Cosine similarity finds articles that are about the same things as the query, and BM25 finds articles that use the same words. A reranker reads the query with each candidate and scores whether the candidate is actually relevant. Pass one the top 50 to 100 candidates from vector_search, bm25_search or hybrid_search:

query = "China semiconductor export controls"
candidates = bilbysearch.vector_search(query, embedded)

reranked = bilbysearch.rerank_local(query, candidates, model_name="BAAI/bge-reranker-v2-m3")
reranked = bilbysearch.rerank_openrouter(query, candidates, model_name="cohere/rerank-v3.5")
reranked = bilbysearch.rerank_jev(query, candidates)

All three take the same arguments - text_col, top_n, min_relevance, and max_chars (default 2,000, since relevance is decided by an article's opening) - and return a copy of the frame sorted by relevance, highest first. Ties keep the incoming order, and the original index is kept. A top_n or max_chars below 1, or a min_relevance outside 0 to 1, is refused before anything is loaded or sent.

	Runs	Needs	Adds
rerank_local	on this machine	the base install	rerank_relevance, rerank_model
rerank_openrouter	on OpenRouter	OPENROUTER_API_KEY	rerank_relevance, rerank_model
rerank_jev	TypeSafe's Jev, through OpenRouter	the [rerank] extra and OPENROUTER_API_KEY	jev_relevance, jev_confidence, jev_probabilities

All three are built on add_relevance_columns, which holds those shared rules: give it a function that scores a list of texts to rerank another way.

Local and OpenRouter rerankers

rerank_local and rerank_openrouter take the reranker by its name in the reranker registry, as the embedding functions take models. There is no default, as no reranker has been chosen for production yet.

bilbysearch.list_local_reranker_models(language="multilingual")
# ['BAAI/bge-reranker-base', 'BAAI/bge-reranker-large', 'BAAI/bge-reranker-v2-m3',
#  'cross-encoder/mmarco-mMiniLMv2-L12-H384-v1', 'maidalun1020/bce-reranker-base_v1',
#  'Qwen/Qwen3-Reranker-0.6B']
bilbysearch.list_openrouter_reranker_models()
bilbysearch.get_reranker_spec("cohere/rerank-v3.5")  # what the registry records

Reranker	Runs	Language	Reads (tokens)
BAAI/bge-reranker-v2-m3	locally	multilingual	8,192
Qwen/Qwen3-Reranker-0.6B	locally	multilingual	40,960
BAAI/bge-reranker-base, BAAI/bge-reranker-large	locally	multilingual (English and Chinese only)	512
maidalun1020/bce-reranker-base_v1	locally	multilingual (English, Chinese, Japanese, Korean)	512
cross-encoder/mmarco-mMiniLMv2-L12-H384-v1	locally (470 MB, fast)	multilingual	512
cross-encoder/ms-marco-MiniLM-L6-v2	locally (90 MB, fast)	English	512
mixedbread-ai/mxbai-rerank-large-v1	locally	English	512
Alibaba-NLP/gte-reranker-modernbert-base	locally	English	8,192
ibm-granite/granite-embedding-reranker-english-r2	locally	English	1,124
cohere/rerank-v3.5	on OpenRouter	multilingual	4,096
cohere/rerank-4-fast, cohere/rerank-4-pro	on OpenRouter	multilingual	32,768
voyageai/rerank-2.5, voyageai/rerank-2.5-lite	on OpenRouter	multilingual	32,000
voyageai/rerank-3, voyageai/rerank-3-lite	on OpenRouter	multilingual	32,000
qwen/qwen3-reranker-8b	on OpenRouter	multilingual	40,960

Both add the same two columns whichever model ran: rerank_relevance, the model's score from 0 to 1, and rerank_model, the model that made it. Code that reads the scores need not know which reranker made them, and a saved frame says whose scores it holds. A second rerank of the same frame replaces both columns together, so to compare two rerankers, run each on the same candidates and join them:

bge = bilbysearch.rerank_local(query, candidates, model_name="BAAI/bge-reranker-v2-m3")
small = bilbysearch.rerank_local(query, candidates, model_name="cross-encoder/ms-marco-MiniLM-L6-v2")
both = bge.join(small[["rerank_relevance"]], rsuffix="_small")

The scores are each model's own. They rank articles for one query and one model. They are not calibrated probabilities, as Jev's are, so a min_relevance tuned for one model means nothing for another. In ex_rerank_local.py, bge-reranker-v2-m3 scores a Chinese article on the query 0.93, while the English-only ms-marco-MiniLM, which cannot read it, scores it 0.00 and the English one 0.999. Use a multilingual reranker on a mixed-language corpus.

Locally, each score is the sigmoid of the model's raw output, applied by rerank_local rather than left to the model, since models disagree about it. The model is loaded on first use and the two most recently used stay loaded: load_reranker loads one ahead of time, and clear_reranker_cache() releases them. A row with no text gets no score and sorts last, and is never shown to the model.

On OpenRouter, set OPENROUTER_API_KEY. Candidates go in batches of batch_size (default 100, one Cohere search) with up to max_concurrent_requests (default 4) in flight, and rate limits and server errors are retried, as for the OpenRouter embeddings. Every candidate is scored - top_n cuts the frame, not the request - and a response that misses a document, or scores one off the 0-1 scale, raises rather than misplacing scores. Cohere charges per search, and Voyage and Qwen per token, which max_chars keeps down. check_openrouter_rerankers.py runs every hosted reranker once, for under a cent.

To add a reranker, add its entry to reranker_config.py: how many tokens it reads (for a local model, measured from the loaded CrossEncoder), its language, and where it runs. For one on OpenRouter, run check_openrouter_rerankers.py too: OpenRouter can list a model that no provider serves, as it listed qwen/qwen3-reranker-0.6b in October 2026, so Qwen3-Reranker-0.6B runs only locally here. Two models that bring their own code - Alibaba-NLP/gte-multilingual-reranker-base and jinaai/jina-reranker-v2-base-multilingual - were considered and left out, because neither runs under transformers 5.

rerank_local.ipynb and rerank_openrouter.ipynb rerank the same candidates with several models, side by side, and rerank_local.ipynb writes a reranker of its own with add_relevance_columns.

Jev

rerank_jev asks TypeSafe's Jev whether each candidate is relevant, and re-sorts by its calibrated probability. It needs the [rerank] extra (see Install). The call goes through bilbystats.batch_jev_api, routed through OpenRouter by default, so it needs an OPENROUTER_API_KEY in the environment. Pass provider="typesafe" or provider="vercel" to route elsewhere, each with its own key.

reranked = bilbysearch.rerank_jev(query, candidates, min_relevance=0.5)
reranked[["title", "jev_relevance", "jev_confidence", "jev_probabilities"]]

jev_relevance is the probability that the article is relevant, jev_confidence is Jev's confidence in that answer and jev_probabilities is the full yes/no dict. Because the probability is absolute, min_relevance works as a cutoff as well as a rank. jev_reranking.ipynb runs vector search then Jev end to end, and sets Jev beside a local reranker.

Evaluating retrieval

Given relevance labels for a query's pool of articles, evaluate scores how well a search ranked them: recall, precision, nDCG and AP at each cutoff.

scores = np.array([0.91, 0.85, 0.80, 0.42, 0.40, 0.12])  # one per article in the pool
labels = np.array([1, 0, 1, 0, 2, 0])  # 0 not relevant, 1 relevant, 2 more relevant

metrics = bilbysearch.evaluate(scores, labels, ks=[2, 5])
# recall@2 0.333, precision@2 0.5, nDCG@2 0.380, MAP@2 0.333,
# recall@5 1.0,   precision@5 0.6, nDCG@5 0.726, MAP@5 0.756   (rounded)

recall_at_k, precision_at_k, ndcg_at_k and ap_at_k compute one metric at one cutoff, and evaluate with no ks reports the cutoffs in DEFAULT_KS: 5, 10, 20 and 50.

• Pass the whole pool. Recall and AP divide by every relevant article in labels, so a pool cut down to a search's top hits loses the relevant articles the search missed, and overstates both.
• Graded labels. Recall, precision and AP count an article as relevant when its label is above 0, each article once. nDCG uses the label itself as the gain, so ranking a label-2 article above a label-1 one scores higher. Labels must be integers: float labels are refused, since they usually mean the scores and labels were passed the wrong way round.
• Ties rank in pool order. numpy's default sort orders equal scores differently from one machine to another, and BM25 scores most of a pool 0, so recall, precision and AP rank tied articles in the order they were passed, the earlier first. nDCG averages the gain over each group of tied articles instead, as scikit-learn computes it.
• AP@k divides by every relevant article in the pool, not just those in the top k, as TREC's MAP@k does, so it falls short of 1 when the pool holds more relevant articles than k. The MAP@k key holds one query's AP: average it over queries for MAP.
• precision@k divides by k even when the pool is smaller than k.
• A pool with no relevant article scores 0 on every metric.

ex_evaluate.py builds a frame of results with one row per query and averages it into MAP; the other scripts in its folder show each metric on its own.

Querying the entities database

entity_profiles.entities holds the people entity_profiles tracks - heads of state, ministers, central bank governors - one row per entity, in a Postgres database on Railway. query_entities_database runs a SQL query on it and returns the rows as a DataFrame.

It connects to the URL in ENTITIES_POSTGRES_URL. Use the Railway Postgres service's DATABASE_PUBLIC_URL: its DATABASE_URL resolves only inside Railway.

export ENTITIES_POSTGRES_URL=postgresql://...

chinese_entities = bilbysearch.query_entities_database(
    """
    SELECT name_en, name_zh, standardised_role
    FROM entity_profiles.entities
    WHERE country = 'People''s Republic of China'
    ORDER BY name_en
    """
)

• Read-only. Each call opens a read-only connection and sends one statement, so Postgres refuses writes - and SQL holding more than one statement - rather than applying them. This guards against mistakes, not intent: the URL's credentials can still write from any other client, so a read-only Postgres role is the real lock.
• Values from variables go in params, not into the SQL with an f-string, so a value can never be read as SQL - help(bilbysearch.query_entities_database) shows how. A value typed into the SQL, as above, doubles any apostrophe: 'People''s'.
• jsonb columns - roles, aliases, sources - come back as Python lists and dicts.

tutorials/entities/entities_table.ipynb describes every column and lists the countries the table holds.

Any other Postgres database

query_entities_database is built on query_postgres, which queries any Postgres the same way - read-only, one statement, values through params - given its URL. Keep each database's URL in an environment variable of its own: read_postgres_url reads it by name, and says which variable to set when it is missing.

export RAILWAY_DATABASE_URL=postgresql://...

database_url = bilbysearch.read_postgres_url("RAILWAY_DATABASE_URL")
tables = bilbysearch.query_postgres(
    """
    SELECT table_schema, table_name
    FROM information_schema.tables
    WHERE table_schema NOT IN ('pg_catalog', 'information_schema')
    """,
    database_url=database_url,
)

database_url has no default, so which database a query reads is always written at the call. An empty URL is refused before connecting, rather than left to libpq, which would fill it in from its own defaults and connect to whatever database is local. examples/postgres/ runs both functions.

Development

uv sync --all-extras
uv run pytest
uv run ruff check . && uv run ruff format .

With conda instead of uv, install from requirements.txt. It pins every package - the [rerank] extra and the dev tools included - at its version in uv.lock, and installs this checkout as an editable package. Run it from the repo root:

conda create -n bilbysearch python=3.12
conda activate bilbysearch
pip install -r requirements.txt
pytest

requirements.txt is exported from uv.lock, never edited by hand: after changing a dependency, run the uv export command in its header. A test fails until you do.

The tests use stub models and a faked OpenRouter, so the suite runs in under twenty seconds, never downloads a real model and never makes an API call. A few chunking tests check the cuts against e5's real tokenizers, and run only where those are already cached. The Postgres tests start a throwaway local Postgres instead of reaching Railway, and are skipped where Postgres's initdb and pg_ctl are not on PATH.

The package is grouped by job: embedding/ makes the vectors, vector_search/ and keyword_search/ find candidates, hybrid_search/ combines them, reranking/ re-sorts them and entities/ reads the entities database, with the modules they share - model_lookup.py, openrouter_client.py, postgres.py and storage.py - directly in bilbysearch/. tests/ and examples/ mirror that layout: the tests for bilbysearch/embedding/embed.py are in tests/embedding/test_embed.py, and the examples of its functions are in examples/embedding/embed/, one ex_<function>.py each. tutorials/ follows it too, with each notebook beside the part it teaches, and tests/test_tutorials.py fails on any file a tutorial - or this README - names that is not there. Dependencies point one way: the searches and rerankers build on embedding/, which builds on the shared modules, hybrid_search/ builds on the two searches, and entities/ builds on postgres.py alone. tests/test_dependency_direction.py records which part may import which, and fails on an import that points the other way, or on a new module until its place there is decided.

Licence

MIT — see LICENSE.