Cached Historical Data Fetcher
Documentation: https://cached-historical-data-fetcher.readthedocs.io
Source Code: https://github.com/34j/cached-historical-data-fetcher
Python utility for fetching any historical data using caching. Suitable for acquiring data that is added frequently and incrementally, e.g. news, posts, weather, etc.
Installation
Install this via pip (or your favourite package manager):
pip install cached-historical-data-fetcher
Features
- Uses cache built on top of
joblib,lz4andaiofiles. - Ready to use with
asyncio,aiohttp,aiohttp-client-cache. Usesasyncio.gatherfor fetching chunks in parallel. (For performance reasons, only usingaiohttp-client-cacheis probably not a good idea when fetching large number of chunks (web requests).) - Based on
pandasand supportsMultiIndex.
Usage
HistoricalDataCache, HistoricalDataCacheWithChunk and HistoricalDataCacheWithFixedChunk
Override get_one() method to fetch data for one chunk. update() method will call get_one() for each unfetched chunk and concatenate results, then save to cache.
from cached_historical_data_fetcher import HistoricalDataCacheWithFixedChunk
from pandas import DataFrame, Timedelta, Timestamp
from typing import Any
# define cache class
class MyCacheWithFixedChunk(HistoricalDataCacheWithFixedChunk[Timestamp, Timedelta, Any]):
delay_seconds = 0.0 # delay between chunks (requests) in seconds
interval = Timedelta(days=1) # interval between chunks, can be any type
start_index = Timestamp.utcnow().floor("10D") # start index, can be any type
async def get_one(self, start: Timestamp, *args: Any, **kwargs: Any) -> DataFrame:
"""Fetch data for one chunk."""
return DataFrame({"day": [start.day]}, index=[start])
# get complete data
print(await MyCacheWithFixedChunk().update())
day
2023-09-30 00:00:00+00:00 30
2023-10-01 00:00:00+00:00 1
2023-10-02 00:00:00+00:00 2
See example.ipynb for real-world example.
IdCacheWithFixedChunk
Override get_one method to fetch data for one chunk in the same way as in HistoricalDataCacheWithFixedChunk.
After updating ids by calling set_ids(), update() method will call get_one() for every unfetched id and concatenate results, then save to cache.
from cached_historical_data_fetcher import IdCacheWithFixedChunk
from pandas import DataFrame
from typing import Any
class MyIdCache(IdCacheWithFixedChunk[str, Any]):
delay_seconds = 0.0 # delay between chunks (requests) in seconds
async def get_one(self, start: str, *args: Any, **kwargs: Any) -> DataFrame:
"""Fetch data for one chunk."""
return DataFrame({"id+hello": [start + "+hello"]}, index=[start])
cache = MyIdCache() # create cache
cache.set_ids(["a"]) # set ids
cache.set_ids(["b"]) # set ids again, now `cache.ids` is ["a", "b"]
print(await cache.update(reload=True)) # discard previous cache and fetch again
cache.set_ids(["b", "c"]) # set ids again, now `cache.ids` is ["a", "b", "c"]
print(await cache.update()) # fetch only new data
id+hello
a a+hello
b b+hello
id+hello
a a+hello
b b+hello
c c+hello
Metadata
Release files for cached-historical-data-fetcher 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| cached_historical_data_fetcher-1.0.0.tar.gz | 13.2 kB | With provenance | Details for cached_historical_data_fetcher-1.0.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| cached_historical_data_fetcher-1.0.0-py3-none-any.whl | Python 3 | none | any | Details for cached_historical_data_fetcher-1.0.0-py3-none-any.whl |
Total release size: 26.2 kB
Release files / cached_historical_data_fetcher-1.0.0.tar.gz
| Download URL | cached_historical_data_fetcher-1.0.0.tar.gz |
|---|---|
| Size | 13.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
80ad1dd969442584746bb1b8dda8ff63c29ba8613d4c7a494b2ddae57caf5a0a
|
|
BLAKE2b-256 checksum How to use checksums |
4f53dafabae481c622d539339d2aee835f9c865892012dcf31a89f00faf9640e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.13
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.
Transparency logRelease files / cached_historical_data_fetcher-1.0.0-py3-none-any.whl
| Download URL | cached_historical_data_fetcher-1.0.0-py3-none-any.whl |
|---|---|
| Size | 13.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
dfe765d8b15955c94046d1162914a354ae52d42e33af2ab0da7140aeb62f6aad
|
|
BLAKE2b-256 checksum How to use checksums |
c2c5e752102d18dc0df7e2a2ce220ffc37e98cd13c6941a9d2ccba182d3df5e0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.13
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.
Transparency logSecurity for cached-historical-data-fetcher 1.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for cached_historical_data_fetcher-1.0.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.