DataCache

Download a dataset once, verify it, and reuse the local file on later runs. DataCache handles streaming downloads, gzip/ZIP decompression, tar-tree installation, offline inspection and SQLite caches built from pandas DataFrames, for Python applications including OpenVax libraries such as pyensembl.

Install

Python 3.11 or newer is required; CI tests Python 3.11–3.14.

python -m pip install datacache
# Optional HTML-table conversion:
python -m pip install "datacache[html]"

Quickstart

Download a file once, then reuse its local path on later calls:

from datacache import Cache

cache = Cache("my-project")
url = "https://raw.githubusercontent.com/openvax/datacache/master/LICENSE"
path = cache.fetch(url, filename="LICENSE", timeout=30)
print(path)

# Reuses the cached file without a network request, even in a later process.
assert cache.fetch(url, filename="LICENSE") == path

Replace the URL and filename with your dataset. Existing files are reused until you refresh them with force=True, or periodically with expire_after; DataCache doesn't ask the server whether a file has changed. Add show_progress=True to display a download bar; no extra installation is needed.

Guides

The guides are also published at https://openvax.github.io/datacache/.

• Working offline example, verification and interface choices
• Downloads, inspection, integrity, freshness, retries and resumption
• Tables, SQLite and transformations
• Shared caches
• Progress and logging
• Pinned bundles, archive trees and the fixed-path file registry
• Interface selection and guarantees
• Derived-artifact materialization: caller-owned builders, outputs published together as one bundle, dependency receipts and resumable inputs.
• Complete API reference
• Individual HTTP ZIP members: bounded range downloads with compressed and expanded checksums.
• Downstream integration: contracts for consuming libraries.
• Release notes and release procedure.

Dependencies and progress display

tqdm is installed by default; existing download APIs enable displays with show_progress=True, while materialize enables them by default. Quiet use does not import tqdm or configure your application's logging. The old datacache[progress] install syntax remains supported as a compatibility alias.

The existing pandas dependency range is unchanged; upgrading DataCache does not introduce a pandas 1.5 requirement. CI covers pandas 1.4.4, 1.5.3, and current releases on supported Python versions.

Development

python -m pip install -e ".[test]"
./lint-and-test.sh
python -m examples.basic_usage

Tests use local files, mocked responses, and local HTTP servers. They do not depend on external dataset servers. See CI for the Python, dependency, and operating-system combinations exercised.

Build the site with python -m pip install -r requirements-docs.txt and ./docs.sh.

License

Apache 2.0; see LICENSE.