Skip to main content

DataCache

Tests Coverage PyPI

Download a dataset once, verify it, and reuse the local file on later runs. DataCache handles streaming downloads, gzip/ZIP decompression, tar-tree installation, offline inspection and SQLite caches built from pandas DataFrames. It is the download layer of the OpenVax tools: PyEnsembl, pepdata, orfdata and Vaxrank use it, and any Python application can.

Install

Python 3.11 or newer is required; CI tests Python 3.11–3.14.

python -m pip install datacache
# Optional HTML-table conversion:
python -m pip install "datacache[html]"

Quick start

Download a file once, then reuse its local path on later calls:

from datacache import Cache

cache = Cache("my-project")
url = "https://raw.githubusercontent.com/openvax/datacache/master/LICENSE"
path = cache.fetch(url, filename="LICENSE", timeout=30)
print(open(path).readline().strip())

# Reuses the cached file without a network request, even in a later process.
print(cache.fetch(url, filename="LICENSE") == path)
Apache License
True

path is under the platform cache directory, for example ~/Library/Caches/my-project/LICENSE on macOS or ~/.cache/my-project/LICENSE on Linux. Replace the URL and filename with your dataset. Existing files are reused until you refresh them with force=True, or periodically with expire_after; DataCache doesn't ask the server whether a file has changed. Add show_progress=True to display a download bar; no extra installation is needed.

Documentation

The documentation starts with an offline example that verifies a file's checksum, then covers one task per page. In reading order:

  1. Download and inspect files: exact destinations, integrity checks, freshness, retries, resumable transfers and offline inspection.
  2. Read tables and build SQLite caches from CSV files and pandas DataFrames.
  3. Share a cache between users and packages.
  4. Install pinned bundles of files that belong together, or archive trees.
  5. Build derived files from downloads and publish them together.
  6. Integrate a consuming library.

Also see choose an interface, the API reference and the release notes.

Dependencies and progress display

tqdm is installed by default; existing download APIs enable displays with show_progress=True, while materialize enables them by default. Quiet use does not import tqdm or configure your application's logging. The old datacache[progress] install syntax remains supported as a compatibility alias.

The existing pandas dependency range is unchanged; upgrading DataCache does not introduce a pandas 1.5 requirement. CI covers pandas 1.4.4, 1.5.3, and current releases on supported Python versions.

Development

python -m pip install -e ".[test]"
./lint.sh   # ruff
./test.sh   # pytest with coverage
python -m examples.basic_usage

The development wrappers use python from your active environment. Set PYTHON=/path/to/python to select another interpreter. Pass pytest arguments through ./test.sh -k test_name or ./lint-and-test.sh -k test_name.

Tests use local files, mocked responses, and local HTTP servers. They do not depend on external dataset servers. See CI for the Python, dependency, and operating-system combinations exercised.

Build the site with python -m pip install -r requirements-docs.txt and ./docs.sh. See the release procedure.

License

Apache License 2.0

Metadata

Release files for datacache 1.31.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datacache 1.31.0
File Size Uploaded Provenance Actions
datacache-1.31.0.tar.gz 305.4 kB Without provenance Details for datacache-1.31.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datacache 1.31.0
File Interpreter ABI Platform Actions
datacache-1.31.0-py3-none-any.whl Python 3 none any Details for datacache-1.31.0-py3-none-any.whl

Total release size: 422.9 kB

Release files / datacache-1.31.0.tar.gz

Download URL datacache-1.31.0.tar.gz
Size 305.4 kB
Tags Source
SHA-256 checksum
How to use checksums
c8b709a2686facfc5edef54ff30f88a445d49a73b2d1b4ac098d7d8ab4de202b
BLAKE2b-256 checksum
How to use checksums
f0688511e7c5d7cc3d22252d684b72334d7f43be9b8029e955afaa20da5841fb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.10

Release files / datacache-1.31.0-py3-none-any.whl

Download URL datacache-1.31.0-py3-none-any.whl
Size 117.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
659535e9aa5f81a2d67ce1029fa9d622c31ce75d79b27cdd9b2187ea2f7ef879
BLAKE2b-256 checksum
How to use checksums
59374c24cc047d40849db14a24a44bbf877b885a10c27f1e3b77d124686dfc95
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.10

Security for datacache 1.31.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.31.0 This release

2 release files No provenance

0.4.7

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page