DataCache
Helpers for transparently downloading datasets
API
- fetch_file(download_url, filename=None, decompress=False, subdir=None)
- fetch_and_transform(transformed_filename, transformer, loader, source_filename, source_url, subdir=None)
- fetch_fasta_dict(download_url, filename=None, subdir=None)
- fetch_fasta_db(table_name, download_url, fasta_filename=None, key_column = 'id', value_column='seq', subdir=None)
- fetch_csv_db(table_name, download_url, csv_filename=None, subdir=None, **pandas_kwargs)
Verified downloads
Use destination to install a single file at an exact path, including its
filename. Supply trusted integrity metadata to check both cached files and new
downloads:
from datacache import fetch_file, validate_file, FileValidationError
path = fetch_file(
"https://example.org/releases/v1/records.tsv.gz",
destination="/data/references/v1/records.tsv",
decompress=True,
expected_sha256=release_metadata["installed_sha256"],
expected_size=release_metadata["installed_size"],
timeout=30,
)
# Read-only validation: no requests, directory creation, locks, or repair.
validate_file(path, expected_sha256=release_metadata["installed_sha256"])
destination accepts a string or pathlib.Path and cannot be combined with
filename or subdir. Existing calls using the default cache continue to work
and can also supply expected_sha256 and expected_size. Parent directories are
created only when fetching a missing file or explicitly refreshing it. A valid
cached file can be reused offline in a readable, non-writable installation.
Both expectations always describe installed bytes, after decompression or
HTML-to-CSV conversion. They do not describe HTTP wire bytes or a compressed
archive when its contents are being installed. To verify and retain an archive,
keep its .gz or .zip suffix at the destination and leave decompress=False.
With an inferred filename, archives are retained by default, including URLs
with query strings or fragments; their existing cache keys are preserved.
decompress=True uses a distinct key for the decompressed contents, keeping the
full URL in the key's digest. With an explicit filename or destination, a
missing compression suffix still implies decompression for compatibility.
decompress=True explicitly requests decompression while preserving an explicit
destination's exact name. Format detection uses the URL path's actual extension
(case-insensitively), independently of query strings and fragments.
For ZIP files, the member matching the output filename is selected, falling
back to the largest non-directory member. No archive paths are extracted.
HTML-to-CSV conversion requires an explicit filename or destination ending
in .csv. Query strings and fragments in inferred cache keys never request
conversion; those downloads retain their original HTML bytes.
A size or SHA-256 mismatch raises FileValidationError, with the path and
expected/actual values in its message. A corrupt cache hit does not trigger a
download: call fetch_file(..., force=True) to explicitly attempt replacement.
validate_file raises FileNotFoundError for missing files and propagates
permission errors; it rejects non-regular files. Fetching propagates transport,
decompression, and filesystem errors so applications can translate them.
Expectations are optional; omitting them provides no integrity guarantee.
Downloads and transformed output use unique staging files in the destination
directory. Only a complete, validated file is published, using os.replace.
Transfer, transformation, validation, or publication failure leaves an existing
destination unchanged and cleans up staging files, including on a handled
keyboard interruption. This avoids shutil.move's cross-filesystem copy and
metadata fallback (related to #39);
SELinux policy compatibility still needs testing on the target installation.
Download and conversion staging files remain owner-only throughout writing and
validation. On replacement, existing files' read/write/execute permission bits
are applied to the validated staging file immediately before publication.
New HTML-to-CSV outputs use
normal file-creation permissions (0666 filtered by the process umask); new
raw or decompressed downloads retain the existing owner-only default. An empty,
disposable file measures normal creation permissions without reading or changing
umask globally; that file never contains downloaded or converted data.
Permission-setting failure preserves the old file and cleans up
staging files. Atomic replacement creates a new inode: ownership, ACLs, and
extended attributes of an existing destination are not copied, and special
setuid/setgid/sticky bits are not preserved.
The publication guarantee assumes a local filesystem supporting atomic replacement of sibling files. Concurrent fetches use separate staging files; the last successful replacement wins and readers opening the destination see complete files. Callers sharing a destination should use the same expectations. A returned path is not a permanent snapshot: later fetches may replace its contents. Platforms that deny replacing an open file may reject publication; the old file is preserved. This does not provide multi-file transactions, distributed coordination, or durability/recovery after power loss or an unhandled process termination, which may leave staging files behind.
5 maintainers
Release files for datacache 1.8.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| datacache-1.8.0.tar.gz | 36.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| datacache-1.8.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 63.5 kB
Release files / datacache-1.8.0.tar.gz
| Download URL | datacache-1.8.0.tar.gz |
|---|---|
| Size | 36.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fc0d3f5ad571490c15810d8b23ae8f712fa0e7cad2c578714e4a5fd6e916b4f5
|
|
BLAKE2b-256 checksum How to use checksums |
0fbb4ac1c353cf4e8448c69dbe4a62bfe16fe407190ae87a6883b9f07d2e0532
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.6
|
Release files / datacache-1.8.0-py3-none-any.whl
| Download URL | datacache-1.8.0-py3-none-any.whl |
|---|---|
| Size | 27.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2fa52ef670ccb6206884532b00a3a854683183a389f076ebb4fd0df81353eb2e
|
|
BLAKE2b-256 checksum How to use checksums |
2680b4131dd47a8ce4d61872779a217034dd0a0c8b8136e21821e5e722eab60e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.6
|
