readerkit

Cached HTTP sessions, cache-directory resolution, and a bulk artifact cache for data readers.

Every cache here has an off switch, and a refresh reaches every readerkit cache in the stack whatever the intervening APIs offer.

Installation

readerkit reaches PyPI with its first release. Until then, install it from the repository.

uv add git+https://github.com/ONEcampaign/readerkit

readerkit requires Python 3.12 or later.

Usage

Cache-directory resolution

resolve_cache_dir turns a kwarg, an environment variable, or platformdirs into an absolute, per-app, per-version cache root. Every other surface below takes its cache_dir from here.

from readerkit import resolve_cache_dir

cache_dir = resolve_cache_dir(app="imf-reader", app_version="1.5.2")
# .../v1/imf-reader/1.5.2, under $IMF_READER_CACHE_DIR, $BBLOCKS_CACHE_DIR, or
# platformdirs.user_cache_dir("readerkit"), in that order.

resolve_cache_dir only resolves a path. To switch caching off, pass cache_dir=None to the surfaces that consume it.

HTTP sessions

build_session returns a configured requests session with bounded retries and jittered backoff, an enforced default timeout, one redirect policy, optional per-host rate limiting, and HTTP response caching when you give it a cache directory.

from readerkit import build_session

session = build_session(app="imf-reader", cache_dir=cache_dir)
response = session.get("https://sdmxcentral.imf.org/...")

Pass cache_dir=None to switch the response cache off. Retries, timeout, pooling and rate limiting stay identical.

Build a session inside each multiprocessing worker process. A session built before a fork shares its socket descriptors with the child, so a response can be delivered to the wrong process. Every session carries a fork guard that raises SessionForkError the first time it is used from a process other than the one that built it.

Bulk artifact cache

ArtifactCache caches large binary payloads (zips, parquet files, whatever a fetcher writes) under per-entry sidecar metadata, with per-key locking and TTL-based staleness.

from datetime import timedelta

from readerkit import ArtifactCache, bulk_fetcher

cache = ArtifactCache(cache_dir=cache_dir, namespace="weo_sdmx")
path = cache.ensure(
    "weo_2026_04",
    fetcher=bulk_fetcher("https://.../weo_2026_04.zip", session=session),
    ttl=timedelta(days=60),
    version="4",
)

Pass cache_dir=None to switch the cache off. Every call then downloads fresh into a scratch directory owned by the cache instance, and managed cache directories are untouched. Call close(), or use the cache as a context manager, to clean the scratch directory up.

with ArtifactCache(cache_dir=None, namespace="weo_sdmx") as cache:
    path = cache.ensure("weo_2026_04", fetcher=bulk_fetcher(url, session=session))

Cache keys

cache_key and cache_key_for_call derive a deterministic cache key from a function's fully bound arguments, so a defaulted parameter stays in the key. oda-reader once hand-listed its key components and left pre_process out, and two different preprocessing options returned identical, wrong data.

from readerkit import cache_key_for_call


def read_gdp(*, country: str, start_year: int, pre_process: bool = True):
    key = cache_key_for_call(
        read_gdp, country=country, start_year=start_year, pre_process=pre_process
    )
    ...

Two calls that differ only in pre_process produce different keys, whether the caller passed it or relied on the default. Use the lower-level cache_key(parts=...) where you assemble the key parts yourself, for example from a URL and a schema version.

Forcing a refresh through layered caches

refresh_scope() forces one refresh of every readerkit-cached artifact and response touched inside the block. It reaches any cache built on readerkit, however deep.

import readerkit

with readerkit.refresh_scope():
    df = (
        pydeflate.read_weo()
    )  # refreshes pydeflate's own artifact, and imf-reader's underneath

refresh_scope() takes effect only for the duration of the with block.

A cached session also accepts refresh= and force_refresh= directly, for a bare call outside any refresh_scope(). refresh=True sends a conditional request, so a 304 returns the cached body unchanged. force_refresh=True issues a new request every time and overwrites whatever was cached. refresh_scope() upgrades to force_refresh, so a caller asking for fresh data gets a new response even from a server whose validators are unreliable.

session.get(url, refresh=True)  # revalidate, may reuse the cached body
session.get(url, force_refresh=True)  # always hits the server, overwrites the cache

Error handling

Every exception readerkit raises subclasses ReaderkitError. Catch that one name to handle anything from the library at once, or catch a specific subclass to handle one failure mode:

• ConfigurationError for invalid arguments, raised eagerly at call time before any I/O.
• CacheDirectoryError when a cache directory cannot be resolved, created, or written to.
• ArtifactCacheError and its subclasses (ArtifactWriteError, ArtifactCorruptError, CacheLockTimeout, CacheLockUnavailable) for failures from ArtifactCache.
• TransportError and its subclasses (SessionForkError, RedirectPolicyError, TruncatedDownloadError) for failures from a build_session session.

Each carries an is_retryable class attribute, so a caller can branch on exc.is_retryable:

from readerkit import ReaderkitError

try:
    path = cache.ensure(key, fetcher=fetcher)
except ReaderkitError as exc:
    if exc.is_retryable:
        path = cache.ensure(key, fetcher=fetcher, refresh=True)
    else:
        raise

HTTP status errors pass through unwrapped. Call raise_for_status() yourself and handle requests.HTTPError, which keeps the status code intact.

Development

readerkit uses uv for dependency management. From a checkout:

uv sync --group dev

Running tests

uv run pytest

Code quality

uv run ruff check .
uv run ruff format .
uv run ty check src/readerkit

Building

uv build

Pre-commit hooks

Pre-commit hooks run on every commit once pre-commit install has been run. To run them across the whole tree:

pre-commit run --all-files

License

readerkit is licensed under the MIT License. See the LICENSE file for details.