timeseries-cache

A persistent SQLite cache for time-series API data. It fetches only missing UTC date ranges and reuses saved rows across runs. The package itself uses only the Python standard library.

Install

Requires Python 3.10 or newer.

uv add timeseries-cache

Use the package

Declare a Schema with a timezone-aware datetime time dimension, then create a Cache and call load(selection, start, end, fetcher). The fetcher receives (start, end, refresh) and returns rows with the declared dimensions and a value. load returns rows, a status (live, cache, stale, or unavailable), and any missing ranges. Pass refresh=True to fetch a range again. Cache files are stored in the directory passed to Cache.

Here is a runnable example using generated hourly observations. Replace the fetcher's body with your API call and return rows in the same format.

from datetime import datetime, timedelta, timezone

from timeseries_cache import Cache, Schema

schema = Schema({"timestamp": datetime, "sensor": str}, "timestamp")
cache = Cache("data/cache", schema)
selection = {"sensor": "A"}


def fetch(start, end, refresh):
    timestamp = start
    while timestamp < end:
        yield {"timestamp": timestamp, **selection, "value": 12.5}
        timestamp += timedelta(hours=1)


start = datetime(2024, 1, 1, tzinfo=timezone.utc)
end = start + timedelta(days=1)

first = cache.load(selection, start, end, fetch)
second = cache.load(selection, start, end, fetch)
print(first.status, second.status, len(second.rows))
# With an empty cache: live cache 24

The file data/cache/timeseries_cache.sqlite persists across Python sessions. Requesting a larger interval fetches only the uncovered parts. read() returns stored rows without fetching; missing() returns uncovered intervals.

Always inspect result.error and result.missing_ranges when a load is not successful. A failed fetch can still return previously cached rows.

Public API

Import Cache, Schema, and CacheResult from timeseries_cache.

CacheResult contains rows, status, missing_ranges, profile_id, fetched_ranges, and error. Rows are ordinary dictionaries. Modifying returned rows does not change stored data. Coverage record dates are ISO 8601 strings; row dates and missing-range bounds are UTC datetime objects.

Schema and row types

Dimensions must include the time field and at least one other field. Their names must be identifiers made from ASCII letters, digits, and underscores, starting with a letter or underscore. value is reserved for the observation.

The value field also accepts None, regardless of value_type. Convert API timestamp strings to aware datetimes before returning rows. Each timestamp must be unique within a selection; represent multiple metrics as a metric dimension and load each metric separately. Nested containers follow JSON semantics, so tuples inside them become lists on reading.

Cache contract

• One selection fixes every non-time dimension. For multiple series, call load for each selection.
• A fetcher must return the complete response for [start, end), including all pages/chunks, or raise. An empty iterable means the interval was checked successfully. Missing observations alone cannot reveal a failed download.
• Every row contains exactly the schema dimensions and value. Timestamps must be timezone-aware. Missing values use None; NaN and infinity are rejected. The start boundary is included; the end boundary is excluded.
• Include inputs that affect results in options, such as provider URL, aggregation or model version. Use separate directories for separate accounts. The cache cannot identify which API a callback uses. JSON dictionaries must have string keys at every nesting level.
• refresh=True reloads the whole interval. max_age=timedelta(hours=1) reloads only missing or expired intervals. Either option passes True to the fetcher when fetching, allowing it to bypass upstream caches.
• If fetching cannot proceed, covered data returns stale; uncovered intervals return unavailable. missing_ranges describes absent coverage, independent of age. Successful zero-row responses still count as coverage.

Errors and freshness

Invalid schemas, selections, periods, and load arguments raise ValueError. replace raises for invalid rows or failed writes; a failed replacement leaves the previous interval intact. load captures exceptions from the fetcher, row validation, and replacement in result.error, stops fetching further gaps, and returns the available data. Earlier successful gaps remain saved. Database failures while opening or reading still raise. KeyboardInterrupt and SystemExit propagate.

Use result.error is not None to check for failure: an exception can have an empty message. Coverage means a range was fetched successfully, not that every possible timestamp has an observation. missing_ranges therefore cannot detect missing samples inside a successful response.

The age of coverage starts after the response has been fully consumed. max_age must be a nonnegative timedelta; zero requests a refetch of existing coverage. Expiry affects reuse without deleting rows. A failed refresh can still return those rows with status stale.

Storage and concurrency

By default, SQLite stores observations in cache_rows and fetched intervals in cache_ranges. Setting Schema.partition_dimension creates separate pairs of tables for each selected partition value. Each record is scoped by schema and profile identifiers derived from the schema, selection, and options. Existing 0.1.x database files can be reopened without migration.

Connections are opened and closed per operation; there is no close() call to manage. SQLite uses WAL mode and a 30-second lock timeout. Each replacement commits rows and coverage together. Readers and writers can use the same local database, but concurrent loads may fetch the same gap more than once. For overlapping replacements, the last committed write wins. A complete load is several operations, not a single database snapshot.

Use a local filesystem for the database. Responses are validated in memory before writing, and results are returned as a list, so split very large requests into smaller intervals. Files persist until you remove them; expiry does not limit disk usage.

Development

uv sync --locked --extra test
uv run --no-sync python -m pytest -q
uv run --no-sync ruff check .
uv run --no-sync ruff format --check .
uv run --no-sync mypy

Tests use temporary databases and mocked fetchers. No credentials or live API requests are needed. The CI workflow checks installed packages on Python 3.10–3.15 and on Linux, macOS, and Windows, using uv's GitHub Actions integration.

Build the wheel and source archive with:

uv build --no-sources

The version is defined in src/timeseries_cache/__init__.py; Hatch reads it when building. Record release changes in CHANGELOG.md. The wheel includes type information and the MIT license; tests ship in the source archive.

License

Released under the MIT License. The full license text is included in the LICENSE file.