arctic-incr-cache

ArcticDB-backed time series cache with incremental updates.

First call fetches the full window from your data source and stores it in ArcticDB. Subsequent calls only fetch the gap between the cached tail and the requested end — then merge and upsert. Incomplete (still-updating) bars are automatically excluded from storage so they never overwrite finalised data.

Install

pip install arctic-incr-cache
# or
uv add arctic-incr-cache

Quick start

import datetime
import arcticdb as adb
from zoneinfo import ZoneInfo
from arctic_incr_cache import IncrCache

arctic = adb.Arctic("lmdb://data/arcticdb")
lib = arctic.get_library("ohlcv-1d", create_if_missing=True)

cache = IncrCache(
    lib,
    fetch=lambda symbol, end, count: your_api.get_daily_bars(symbol, end=end, count=count),
    get_tz=lambda symbol: ZoneInfo("America/New_York"),
)

df = cache.get("AAPL", end=datetime.date(2024, 6, 1), count=60)

• First call — fetches 60 bars from your API, stores in ArcticDB, returns.
• Second call (same or later end) — serves from ArcticDB; fetches only the gap if the cache is stale.

Intraday data

Set bar_minutes to the bar width and provide get_tz to return the market timezone:

from zoneinfo import ZoneInfo

intraday = IncrCache(
    lib,
    fetch=lambda symbol, end, count: your_api.get_minute_bars(symbol, end=end, count=count),
    bar_minutes=1,
    default_count=390 * 5,
    get_tz=lambda symbol: ZoneInfo("America/New_York"),
)

Concurrency

By default writes run in a daemon thread. Pass spawn and lock_class for gevent or other async runtimes:

import gevent
import gevent.lock

cache = IncrCache(
    lib,
    fetch=my_fetch,
    spawn=gevent.spawn,
    lock_class=gevent.lock.BoundedSemaphore,
)

Timezone handling

When get_tz returns a timezone for a symbol:

• fetch return — must be tz-aware. Timestamps are converted to the configured market timezone internally.
• Storage — data is stored in ArcticDB as tz-aware in the configured timezone.
• Return — get() returns a tz-aware DataFrame in the configured timezone.
• end parameter — date becomes end-of-day in market timezone; naive datetime is interpreted as local timezone, then converted; tz-aware is converted directly.

Interval convention

end is a bar timestamp (a point), not a range boundary.

• Filter — closed: index <= end. fetch() must follow the same rule (start <= ts <= end); a strict < silently drops bar@end.
• Freshness:
  • Daily — closed: last.date() >= end.date().
  • Intraday — right-open: last >= end - bar_width. An intraday bar at t covers [t, t+bar_width), so bar@end doesn't exist at session boundaries (e.g. 16:00 close, 20:00 POST end).

A still-updating bar (now daily; within bar_width intraday) counts as one bar older for freshness — mirroring its exclusion from storage.

Index convention

fetch() must return a DataFrame with a DatetimeIndex as the index — this is the time axis for all cache operations (querying, merging, freshness checks). ArcticDB's date_range queries operate on the index, so no column name configuration is needed. If your data source returns time as a regular column, call df.set_index("date") (or similar) inside your fetch function.

Constructor parameters

License

MIT