thds.adls
A high-performance Azure Data Lake Storage (ADLS Gen2) client for the THDS monorepo. It wraps the Azure SDK with hash-aware caching, azcopy acceleration, and shared client/credential plumbing so applications can transfer large blob datasets quickly and reliably.
Highlights
- Environment-aware paths first: Almost every consumer starts by importing
fqn,AdlsFqn, anddefaults.env_root()to build storage-account/container URIs that follow the current THDS environment. - Cache-backed reads:
download_to_cacheis the standard entry point for pulling blobs down with a verified hash so local workflows, tests, and pipelines can operate on read-only copies. - Bulk filesystem helpers:
ADLSFileSystempowers scripts and jobs that need to walk directories, fetch batches of files, or mirror hive tables without re-implementing Azure SDK plumbing. - Spark/Databricks bridges:
abfssanduriconversions keep analytics code agnostic to whether it needs anadls://,abfss://,https://, ordbfs://view of the same path. - Composable utilities: Higher-level modules (cache, upload, copy, list) layer on top of those imports so teams can opt into more advanced behavior without leaving the public API surface.
Key Modules
| Component | Typical usage in the monorepo |
|---|---|
fqn |
Parse, validate, and join ADLS paths; used when materializing model datasets and configuring pipelines. |
AdlsFqn |
Strongly typed value passed between tasks and tests to represent a single blob or directory. |
defaults / named_roots |
Resolve environment-specific storage roots (defaults.env_root(), named_roots.require(...)). |
download_to_cache (cached module) |
Bring a blob down to the shared read-only cache before analytics, feature builds, or test fixtures run. |
ADLSFileSystem (impl module) |
Fetch or list entire directory trees and integrate with caching inside scripts and notebooks. |
abfss |
Translate AdlsFqn objects into abfss:// URIs for Spark/Databricks jobs. |
uri |
Normalize adls://, abfss://, https://, and dbfs:// strings into AdlsFqn values (and vice versa). |
global_client / shared_credential |
Shared, fork-safe Azure clients and credentials backing the public helpers above. |
Example Usage
-
Use the caching helpers and Source integration:
from thds.adls import cached, upload, source cache_path = cached.download_to_cache("adls://acct/container/path/to/file") src = upload("adls://acct/container/path/out.parquet", cache_path) verified = source.get_with_hash(src.uri)
-
For CLI usage, run (from repo root):
uv run python -m thds.adls.tools.download adls://acct/container/path/file
Operational Notes
- Hash metadata: Uploads attach
hash_xxh3_128_b64automatically when the bytes are known. Download completion back-fills missing hashes when permissions allow. - Locks and concurrency: Large transfers acquire per-path file locks to keep azcopy instances
cooperative. Global HTTP connection pools default to 100 but are configurable via
thds.core.config. - Error handling:
BlobNotFoundErrorand other ADLS-specific exceptions translate into custom error types to simplify retries and diagnostics. - Extensibility: Additional hash algorithms can be registered by importing dependent packages (e.g.,
blake3). Named roots can be populated dynamically via environment-specific modules (thds.adls._thds_defaultshook).
1 maintainer
Metadata
1 maintainer
Release files for thds.adls 4.5.20260130200330
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| thds_adls-4.5.20260130200330-py3-none-any.whl | Python 3 | none | any | Details for thds_adls-4.5.20260130200330-py3-none-any.whl |
Release files / thds_adls-4.5.20260130200330-py3-none-any.whl
| Download URL | thds_adls-4.5.20260130200330-py3-none-any.whl |
|---|---|
| Size | 63.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2d00f82f23e5710ac7c7a5d5801c2e453c3aad7c796a9a960407a818ec9bd24a
|
|
BLAKE2b-256 checksum How to use checksums |
16bc29e3f6423b7d63a491f2c6692f4b7511360e617762f010d396e220da5f3b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.3
|
Security for thds.adls 4.5.20260130200330
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
