Cached Historical Data Fetcher
Python utility for fetching any historical data using caching. Suitable for acquiring data that is added frequently and incrementally, e.g. news, posts, weather, etc.
Installation
Install this via pip (or your favourite package manager):
pip install cached-historical-data-fetcher
Features
- Uses cache built on top of
joblib,lz4andaiofiles. - Ready to use with
asyncio,aiohttp,aiohttp-client-cache. Usesasyncio.gatherfor fetching chunks in parallel. (For performance reasons, only usingaiohttp-client-cacheis probably not a good idea when fetching large number of chunks (web requests).) - Based on
pandasand supportsMultiIndex.
Usage
HistoricalDataCache, HistoricalDataCacheWithChunk and HistoricalDataCacheWithFixedChunk
Override get_one() method to fetch data for one chunk. update() method will call get_one() for each unfetched chunk and concatenate results, then save to cache.
from cached_historical_data_fetcher import HistoricalDataCacheWithFixedChunk
from pandas import DataFrame, Timedelta, Timestamp
from typing import Any
# define cache class
class MyCacheWithFixedChunk(HistoricalDataCacheWithFixedChunk[Timestamp, Timedelta, Any]):
delay_seconds = 0.0 # delay between chunks (requests) in seconds
interval = Timedelta(days=1) # interval between chunks, can be any type
start_index = Timestamp.utcnow().floor("10D") # start index, can be any type
async def get_one(self, start: Timestamp, *args: Any, **kwargs: Any) -> DataFrame:
"""Fetch data for one chunk."""
return DataFrame({"day": [start.day]}, index=[start])
# get complete data
print(await MyCacheWithFixedChunk().update())
day
2023-09-30 00:00:00+00:00 30
2023-10-01 00:00:00+00:00 1
2023-10-02 00:00:00+00:00 2
See example.ipynb for real-world example.
IdCacheWithFixedChunk
Override get_one method to fetch data for one chunk in the same way as in HistoricalDataCacheWithFixedChunk.
After updating ids by calling set_ids(), update() method will call get_one() for every unfetched id and concatenate results, then save to cache.
from cached_historical_data_fetcher import IdCacheWithFixedChunk
from pandas import DataFrame
from typing import Any
class MyIdCache(IdCacheWithFixedChunk[str, Any]):
delay_seconds = 0.0 # delay between chunks (requests) in seconds
async def get_one(self, start: str, *args: Any, **kwargs: Any) -> DataFrame:
"""Fetch data for one chunk."""
return DataFrame({"id+hello": [start + "+hello"]}, index=[start])
cache = MyIdCache() # create cache
cache.set_ids(["a"]) # set ids
cache.set_ids(["b"]) # set ids again, now `cache.ids` is ["a", "b"]
print(await cache.update(reload=True)) # discard previous cache and fetch again
cache.set_ids(["b", "c"]) # set ids again, now `cache.ids` is ["a", "b", "c"]
print(await cache.update()) # fetch only new data
id+hello
a a+hello
b b+hello
id+hello
a a+hello
b b+hello
c c+hello
Contributors ✨
Thanks goes to these wonderful people (emoji key):
This project follows the all-contributors specification. Contributions of any kind welcome!
1 maintainer
Metadata
1 maintainer
Release files for cached-historical-data-fetcher 0.2.31
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| cached_historical_data_fetcher-0.2.31.tar.gz | 13.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| cached_historical_data_fetcher-0.2.31-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 27.0 kB
Release files / cached_historical_data_fetcher-0.2.31.tar.gz
| Download URL | cached_historical_data_fetcher-0.2.31.tar.gz |
|---|---|
| Size | 13.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e355bedcc552b8268e1682fc37a87a4bb3ff5372d54feea06bd23c539833b713
|
|
BLAKE2b-256 checksum How to use checksums |
f845908191adb13d638e6b6413184c410dddf571a5e3c8c1623e1e9bce5c1ad0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Release files / cached_historical_data_fetcher-0.2.31-py3-none-any.whl
| Download URL | cached_historical_data_fetcher-0.2.31-py3-none-any.whl |
|---|---|
| Size | 13.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
dd71e853b4ec361261569c29ccb8593d8109a3f1be2073d32b953844fd441c5a
|
|
BLAKE2b-256 checksum How to use checksums |
73c0f780a5b7c17f9a87270f234f70ca58bc567ed633e426ff86f0bbf9993c76
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
