DataCache

Helpers for transparently downloading datasets

API

• fetch_file(download_url, filename=None, decompress=False, subdir=None)
• fetch_and_transform(transformed_filename, transformer, loader, source_filename, source_url, subdir=None)
• fetch_fasta_dict(download_url, filename=None, subdir=None)
• fetch_fasta_db(table_name, download_url, fasta_filename=None, key_column = 'id', value_column='seq', subdir=None)
• fetch_csv_db(table_name, download_url, csv_filename=None, subdir=None, **pandas_kwargs)

Verified downloads

Use destination to install a single file at an exact path, including its filename. Supply trusted integrity metadata to check both cached files and new downloads:

from datacache import fetch_file, validate_file, FileValidationError

path = fetch_file(
    "https://example.org/releases/v1/records.tsv.gz",
    destination="/data/references/v1/records.tsv",
    decompress=True,
    expected_sha256=release_metadata["installed_sha256"],
    expected_size=release_metadata["installed_size"],
    timeout=30,
)

# Read-only validation: no requests, directory creation, locks, or repair.
validate_file(path, expected_sha256=release_metadata["installed_sha256"])

destination accepts a string or pathlib.Path and cannot be combined with filename, subdir, or cache_root. Existing calls using the default cache continue to work and can also supply expected_sha256 and expected_size. Parent directories are created only when fetching a missing file or explicitly refreshing it. A valid cached file can be reused offline in a readable, non-writable installation.

Both expectations always describe installed bytes, after decompression or HTML-to-CSV conversion. They do not describe HTTP wire bytes or a compressed archive when its contents are being installed. To verify and retain an archive, keep its .gz or .zip suffix at the destination and leave decompress=False. With an inferred filename, archives are retained by default, including URLs with query strings or fragments; their existing cache keys are preserved. decompress=True uses a distinct key for the decompressed contents, keeping the full URL in the key's digest. When a download endpoint's inferred filename has no removable archive suffix, .decompressed distinguishes its output. With an explicit filename or destination, a missing compression suffix still implies decompression for compatibility. decompress=True explicitly requests decompression while preserving an explicit destination's exact name. Format detection prefers a supported extension on the URL path (case-insensitively). For download endpoints without a supported path extension, a filename in the final query parameter is also recognized for compatibility, such as IEDB's downloader.php?file_name=doc/data.zip. Bare format hints such as ?format=.gz and fragments do not select a format. For ZIP files, the member matching the output filename is selected, falling back to the largest non-directory member. No archive paths are extracted. HTML-to-CSV conversion requires an explicit filename or destination ending in .csv. Query strings and fragments in inferred cache keys never request conversion; those downloads retain their original HTML bytes.

A size or SHA-256 mismatch raises FileValidationError, with the path and expected/actual values in its message. A corrupt cache hit does not trigger a download: call fetch_file(..., force=True) to explicitly attempt replacement. validate_file raises FileNotFoundError for missing files and propagates permission errors; it rejects non-regular files. Fetching propagates transport, decompression, and filesystem errors so applications can translate them. Expectations are optional; omitting them provides no integrity guarantee.

Downloads and transformed output use unique staging files in the destination directory. Only a complete, validated file is published, using os.replace. Transfer, transformation, validation, or publication failure leaves an existing destination unchanged and cleans up staging files, including on a handled keyboard interruption. This avoids shutil.move's cross-filesystem copy and metadata fallback (related to #39); SELinux policy compatibility still needs testing on the target installation.

Download and conversion staging files remain owner-only throughout writing and validation. On replacement, existing files' read/write/execute permission bits are applied to the validated staging file immediately before publication. New HTML-to-CSV outputs use normal file-creation permissions (0666 filtered by the process umask); new raw or decompressed downloads retain the existing owner-only default. An empty, disposable file measures normal creation permissions without reading or changing umask globally; that file never contains downloaded or converted data. Permission-setting failure preserves the old file and cleans up staging files. Atomic replacement creates a new inode: ownership, ACLs, and extended attributes of an existing destination are not copied, and special setuid/setgid/sticky bits are not preserved.

The publication guarantee assumes a local filesystem supporting atomic replacement of sibling files. Concurrent fetches use separate staging files; the last successful replacement wins and readers opening the destination see complete files. Callers sharing a destination should use the same expectations. A returned path is not a permanent snapshot: later fetches may replace its contents. Platforms that deny replacing an open file may reject publication; the old file is preserved. This does not provide multi-file transactions, distributed coordination, or durability/recovery after power loss or an unhandled process termination, which may leave staging files behind.

Read-only cache inspection

Path lookup, presence, and integrity checks have different contracts:

from datacache import Cache, expected_path, file_exists, inspect_file, inspect_files

root = "/data/references/v1"
path = expected_path(filename="records.tsv", cache_root=root)
present = file_exists(filename="records.tsv", cache_root=root)
result = inspect_file(path, expected_sha256=release_metadata["installed_sha256"])
if result.status == "available" and result.verified:
    process_records(result.path)

# The object API uses the same paths and validation contract.
cache = Cache("references", cache_root=root)
result = cache.inspect(filename="records.tsv",
                       expected_sha256=release_metadata["installed_sha256"])

# Inventory required files using trusted, caller-supplied metadata.
installation = inspect_files(root, {
    "records.tsv": {"expected_sha256": release_metadata["installed_sha256"]},
    "manifest.json": {"expected_sha256": release_metadata["manifest_sha256"]},
})

expected_path, resolve_path, and Cache.local_path(download=False) only compute paths, with no filesystem access. file_exists and Cache.exists check presence without requiring a readable regular file: directories count as present, broken symlinks count as absent, and permission errors propagate. None of these operations creates a directory or a lock file.

inspect_file, inspect_files, and Cache.inspect return results with path, status, verified, and error attributes:

verified is true only when a supplied SHA-256 digest matched; presence, readability, or a size check alone does not establish verified integrity. error retains the original filesystem or validation exception when unavailable. Invalid integrity arguments raise ValueError rather than reporting a cache problem. inspect_files also returns a files mapping of individual results; for example, a missing manifest makes the installation corrupt while that manifest's individual status is missing. Inaccessible files take precedence over corrupt or missing files in the aggregate result. Required names must be normalized relative paths; nested names are supported. Use {} or None for an entry without integrity metadata. The required mapping must be nonempty.

Inspection only reads local files. It never downloads, writes manifests, creates locks, or attempts recovery, so a valid read-only version and a missing sibling version can be inspected independently. The caller supplies required files and trusted integrity metadata; datacache does not parse manifests or discover versions. Symlinks are followed as in ordinary file access. Inventory is not a snapshot across concurrent external changes or a multi-file installation mechanism; versioned bundle installation is tracked in #59.

cache_root accepts a string or pathlib.Path and names the actual directory containing cached files, overriding the platform location selected by subdir. Relative roots remain relative to the current working directory. It is supported by fetch_file, expected_path, file_exists, resolve_path, build_path, and Cache. An exact destination is mutually exclusive with cache_root. Default cache locations and filename normalization remain the same. build_path still creates parents; use resolve_path for pure lookup. Creation and repair remain explicit operations: use fetch_file or Cache.fetch, supplying integrity metadata and force=True for replacement. Cache.fetch validates every reuse and respects different filenames for the same URL. Its database and deletion methods also use the selected cache root. Database paths preserve the filesystem meaning of symlinks followed by ... Cache.delete_all() clears the root's contents while preserving the directory and its permissions, including when the root is . or a symlink. Symlinks inside the cache are removed without clearing their external targets.

Downstream compatibility

The private _download_and_decompress_if_necessary entry point, used by pyensembl, retains its pre-1.8 literal-URL format inference when transform flags are omitted. In particular, query/fragment-bearing archive URLs retain the same bytes under pyensembl's existing cache keys. Explicit transform flags and the public fetch_file API retain the parsed-URL behavior documented above. The IEDB download endpoints used by pepdata continue to decompress archives named at the end of the URL query into the requested CSV filenames. The _decompress_to_file helper remains available and uses failure-safe atomic publication. New integrations should use the public download and inspection APIs.