relib

Small Python utilities for dictionaries, iterables, files, async tasks and local text search. Python 3.10+. The package has no required third-party dependencies.

Install

uv add relib

Most utilities are exported from relib. Import the search classes from relib.search.

from relib import distinct, group, read_json, roll_tasks
from relib.search import SearchIndex

Utilities

Dictionaries and iterables

from relib import distinct, group, partition, pick

assert distinct(["Orion", "Moon", "Orion"]) == ["Orion", "Moon"]
assert group([("clear", "Monday"), ("cloudy", "Tuesday"), ("clear", "Friday")]) == {
    "clear": ["Monday", "Friday"],
    "cloudy": ["Tuesday"],
}
assert pick({"object": "Orion", "exposure": 30, "note": "clear"}, ["object", "exposure"]) == {
    "object": "Orion", "exposure": 30,
}
assert partition((seconds >= 30, seconds) for seconds in [10, 30, 60]) == ([30, 60], [10])

JSON files

from pathlib import Path
from relib import read_json, write_json

path = Path("settings.json")
settings = read_json(path, default={"exposure": 30})
write_json(path, settings, indent=2)

read_json uses the default only when the file is absent. write_json writes to an existing parent directory. clear_directory removes .DS_Store files and empty directories; it does not recursively delete ordinary files.

Async tasks

import asyncio
from pathlib import Path
from relib import as_async, roll_tasks

@as_async(workers=4)
def read_text(path: Path) -> str:
    return path.read_text()

async def main():
    texts = await roll_tasks(
        (read_text(path) for path in Path("notes").glob("*.txt")),
        workers=4,
    )
    print(len(texts))

asyncio.run(main())

as_async runs a synchronous function in a thread pool. roll_tasks limits how many awaitables run concurrently and returns results in input order. async_limit(workers=4) decorates an existing async function with a concurrency limit. measure_duration("name") is a context manager that prints elapsed time.

Progress bars require tqdm; df_from_array requires numpy and pandas. These packages are imported only by the features that use them.

Search dictionaries from disk

SearchIndex builds from JSON-compatible dictionaries. Parsing CSV, XML or other sources belongs to the caller. Records can arrive through a regular one-pass generator; the index does not retain the complete dictionaries in memory.

The examples below use an astronomy observation log.

Build and search

This is a complete runnable example:

import asyncio
from relib.search import SearchIndex

observations = [
    {
        "id": "000021",
        "title": "Orion nebula red 80 mm",
        "description": "Hydrogen emission long exposure",
        "observer": "Anaïs",
        "categories": ["Deep sky", "Deep sky > Nebulae"],
        "aperture": "80mm",
        "image": "images/orion-red.fits",
    },
    {
        "id": "000022",
        "title": "Orion nebula blue 80mm",
        "description": "Oxygen emission long exposure",
        "observer": "Anaïs",
        "categories": ["Deep sky", "Deep sky > Nebulae"],
        "aperture": "80mm",
        "image": "images/orion-blue.fits",
    },
    {
        "id": "000031",
        "title": "Lunar mosaic",
        "description": "High resolution crater survey",
        "observer": "Sam",
        "categories": ["Solar system", "Solar system > Moon"],
        "aperture": "200mm",
        "image": "images/moon.fits",
    },
]

async def main():
    index = await SearchIndex.build(
        "observations.idx",
        iter(observations),
        fields={"id": 8, "title": 4, "observer": 2, "categories": 2, "description": 1, "aperture": 1},
        filter_fields=["observer", "categories"],
    )
    with index:
        for hit in await index.search("anais red 80mm"):
            print(hit.record["id"], hit.record["title"], hit.matched_terms)
            # 000021 Orion nebula red 80 mm ('anais', 'red', '80', 'mm')

asyncio.run(main())

fields selects searchable fields and their relative weights. Passing a list of names instead gives every field weight 1. Searchable values can be scalars or lists. Other fields, such as image, remain in the stored record without being indexed. Keep identifiers as strings to preserve leading zeros.

Each result is a SearchHit with the original record dictionary, a numeric score and normalized matched_terms. Only records on the returned page are decoded, during the search call. hit.record is already a dictionary.

Query options

Reuse an open index across queries. Within an async function:

with SearchIndex("observations.idx") as index:
    by_id = await index.search("000021")
    words = await index.search("hydrogen exposure")
    completion = await index.search("Orion neb", prefix=True)
    category = await index.search(filters={"categories": "Deep sky"})
    filtered = await index.search("nebula", filters={"observer": "anais"})
    next_page = await index.search("nebula", offset=1, limit=1)
    alternatives = await index.search("Orion ultraviolet", match="any")

• All words must match by default, across any indexed fields. Unknown words are not silently removed. match="any" permits partial matches and ranks the number of matched words before relevance.
• Case and accents normalize: anais matches Anaïs. Letters and numbers split, so 80mm matches 80 mm. 000021 and 21 remain different tokens.
• prefix=True completes only the final query word. Punctuation is text separation, not query syntax.
• Filters match complete values, ignoring case and accents. A list of values means OR within one field; separate fields combine with AND. Declare filter fields when building. Category trees have no special meaning: include parent paths in each record if you want to filter by them.
• Empty queries browse filtered records in source order. Without filters, an empty query returns no results. len(index) gives the record count.
• Ranking combines field weights, word frequencies, field lengths and how rare each word is in the collection. Equal scores use source order as the tie-break.
• There is no stemming, automatic translation, typo correction or semantic matching. Add aliases to your input dictionaries when needed.

Storage and lifetime

An index is one file containing records, term/filter postings, record offsets and metadata. Opening it memory-maps the file and reads the term/filter catalog; it does not decode every record. Query caches retain up to 128 text lookups and 128 filter values.

Builds create a new snapshot and replace the target file after success. Failed record streams leave the previous file intact. Existing open readers keep their snapshot; new readers see the replacement. Updates require rebuilding the index.

Building retains the vocabulary and compact postings in memory, so its memory usage still grows with the collection. Broad partial-match queries can inspect many more candidates than selective queries. Concurrent reads are supported; close the index only after its queries finish.