DataCache

Download a dataset once, verify it, and reuse the local file on later runs. DataCache handles streaming downloads, gzip/ZIP decompression, tar-tree installation, offline inspection and SQLite caches built from pandas DataFrames. It is the download layer of the OpenVax tools: PyEnsembl, pepdata, orfdata and Vaxrank use it, and any Python application can.

Install

Python 3.11 or newer is required; CI tests Python 3.11–3.14.

python -m pip install datacache
# Optional HTML-table conversion:
python -m pip install "datacache[html]"

Quick start

Download a file once, then reuse its local path on later calls:

from datacache import Cache

cache = Cache("my-project")
url = "https://raw.githubusercontent.com/openvax/datacache/master/LICENSE"
path = cache.fetch(url, filename="LICENSE", timeout=30)
print(open(path).readline().strip())

# Reuses the cached file without a network request, even in a later process.
print(cache.fetch(url, filename="LICENSE") == path)

Apache License
True

path is under the platform cache directory, for example ~/Library/Caches/my-project/LICENSE on macOS or ~/.cache/my-project/LICENSE on Linux. Replace the URL and filename with your dataset. Existing files are reused until you refresh them with force=True, or periodically with expire_after; DataCache doesn't ask the server whether a file has changed. Add show_progress=True to display a download bar; no extra installation is needed.

Documentation

The documentation starts with an offline example that verifies a file's checksum, then covers one task per page. In reading order:

1. Download and inspect files: exact destinations, integrity checks, freshness, retries, resumable transfers and offline inspection.
2. Read tables and build SQLite caches from CSV files and pandas DataFrames.
3. Share a cache between users and packages.
4. Install pinned bundles of files that belong together, or archive trees.
5. Build derived files from downloads and publish them together.
6. Integrate a consuming library.

Also see choose an interface, the API reference and the release notes.

Dependencies and progress display

tqdm is installed by default; existing download APIs enable displays with show_progress=True, while materialize enables them by default. Quiet use does not import tqdm or configure your application's logging. The old datacache[progress] install syntax remains supported as a compatibility alias.

The existing pandas dependency range is unchanged; upgrading DataCache does not introduce a pandas 1.5 requirement. CI covers pandas 1.4.4, 1.5.3, and current releases on supported Python versions.

Development

python -m pip install -e ".[test]"
./lint.sh   # ruff
./test.sh   # pytest with coverage
python -m examples.basic_usage

Tests use local files, mocked responses, and local HTTP servers. They do not depend on external dataset servers. See CI for the Python, dependency, and operating-system combinations exercised.

Build the site with python -m pip install -r requirements-docs.txt and ./docs.sh. See the release procedure.

License

Apache License 2.0