DataCache
Download, verify, transform, and cache datasets for Python applications, including OpenVax libraries such as pyensembl. DataCache provides streaming downloads, gzip/ZIP decompression, reusable local paths, offline inspection, and SQLite caches built from pandas DataFrames.
Install
Python 3.9 or newer is required.
python -m pip install datacache
# Optional progress bars and HTML-table conversion:
python -m pip install "datacache[progress,html]"
Progress is opt-in with show_progress=True. Normal use does not import tqdm
or configure your application's logging.
The existing pandas dependency range is unchanged; upgrading DataCache does not introduce a pandas 1.5 requirement. CI covers pandas 1.4.4, 1.5.3, and current releases on supported Python versions.
Quickstart
This example runs entirely offline and cleans up after itself:
import gzip
import hashlib
from pathlib import Path
from tempfile import TemporaryDirectory
from datacache import Cache
with TemporaryDirectory() as directory:
root = Path(directory)
contents = b">reference\nACGT\n"
source = root / "reference.fa.gz"
source.write_bytes(gzip.compress(contents))
cache = Cache("references", cache_root=root / "cache")
path = cache.fetch(
source.as_uri(), # HTTP, HTTPS, and FTP URLs also work.
filename="reference.fa",
expected_sha256=hashlib.sha256(contents).hexdigest(),
)
assert Path(path).read_bytes() == contents
source.unlink()
assert cache.fetch(source.as_uri(), filename="reference.fa") == path
print(cache.inspect(filename="reference.fa").status) # available
For real releases, get the expected hash from trusted release metadata. Hashes describe the installed bytes after decompression or conversion. A hash computed from an untrusted download does not establish authenticity.
To install at an exact path instead of using a cache key:
from datacache import fetch_file
path = fetch_file(
"https://example.org/releases/v1/records.tsv.gz",
destination="references/v1/records.tsv",
decompress=True,
timeout=30,
show_progress=True, # Requires datacache[progress].
)
Replace the example URL with your dataset URL. Existing files are reused;
force=True explicitly replaces them. When integrity expectations are supplied,
an invalid cache hit raises FileValidationError instead of silently replacing
the file. Failed downloads leave the previous file intact.
Choose the right API
| Task | API | Result |
|---|---|---|
| Download or reuse one file | fetch_file(...), Cache.fetch(...) |
Local path string |
| Compute a path without filesystem access | expected_path(...), Cache.local_path(...) |
Path string |
| Check presence | file_exists(...), Cache.exists(...) |
Boolean; does not establish integrity |
| Validate bytes, raising on failure | validate_file(...) |
Path string |
| Inspect without repair or network access | inspect_file(...), Cache.inspect(...) |
FileInspection |
| Inspect a set of required files | inspect_files(root, files) |
CacheInspection |
| Explicitly share an existing private file | make_file_readable(...), Cache.make_readable(...) |
Path string; POSIX only |
| Download and parse CSV/TSV | fetch_csv_dataframe(...) |
pandas DataFrame |
| Cache a custom file transformation | fetch_and_transform(...) |
Transformer/loader result |
| Create a SQLite cache | db_from_dataframe(...), db_from_dataframes(...) |
Open SQLite connection |
| Download CSV into SQLite | fetch_csv_db(...) |
Open SQLite connection |
| Reopen a database with matching metadata | connect_if_correct_version(...) |
Connection or None |
The library does not export fetch_fasta_dict or fetch_fasta_db. Download
FASTA files with fetch_file, then parse them in the consuming library.
Guides
- Downloads and cache inspection: destinations, naming, decompression, integrity, retries, concurrency, and downstream compatibility.
- Progress and logging: tqdm, callbacks, retries, and independent download options for CSV helpers.
- SQLite and transformations: numeric fidelity, column names, versioning, rollback, connection ownership, and custom transformations.
- Shared caches: permissions, read-only use, and troubleshooting existing installations.
- Downstream integration: contracts for consuming libraries.
- Release notes and release procedure.
Guarantees and limits
Downloads are staged privately and published atomically after validation. New files respect the process umask; replacements preserve existing access permissions. This includes pyensembl's private download helpers.
Custom single-file transformations publish only successful output. Existing SQLite caches rebuild in a transaction: failure rolls back both schema and rows. New databases are built privately before publication. Cached data is reused by path or database version; DataCache does not automatically discover remote changes or repair previously corrupted caches.
Upgrades keep existing cache names and database metadata compatible. Valid cache hits do not rewrite files, change permissions, or apply new schema constraints. See upgrading existing caches and sharing old private files.
File publication requires local filesystem support for atomic replacement; new SQLite database publication also requires hard links. SQLite locking and transactions govern database rebuilds. These are single-file guarantees, not a multi-file release installer or a distributed lock service.
Development
python -m pip install -e ".[test]"
./lint-and-test.sh
python -m examples.basic_usage
Tests use local files, mocked responses, and local HTTP servers. They do not depend on external dataset servers. See CI for the Python, dependency, and operating-system combinations exercised.
5 maintainers
Release files for datacache 1.10.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| datacache-1.10.0.tar.gz | 81.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| datacache-1.10.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 122.3 kB
Release files / datacache-1.10.0.tar.gz
| Download URL | datacache-1.10.0.tar.gz |
|---|---|
| Size | 81.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3a0d3360e378d2db4e1fba0c5ba2023231e77a4f0447f27f35b2f7475ddceb6f
|
|
BLAKE2b-256 checksum How to use checksums |
c661c05874bffbe9c81843b52fce31ae19e0f4e409972a18810842d10455494b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.6
|
Release files / datacache-1.10.0-py3-none-any.whl
| Download URL | datacache-1.10.0-py3-none-any.whl |
|---|---|
| Size | 40.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
95b6dfc3aaaee26e5c162727a57b021b9d56c2adee19759647bd2acfb6c643bf
|
|
BLAKE2b-256 checksum How to use checksums |
1218d92b360e334ac14f5140456b912aed1e1f2195915703ce05e6ff9b12b299
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.6
|
