ccflow-s3
ccflow models for S3
Overview
ccflow-s3 provides public, domain-neutral S3 and S3-compatible storage callable models for ccflow workflows. It provides client and credential configuration, object reads/writes, existence checks, listing, manifests, atomic publish behavior, an S3-backed cache adapter, a generic artifact store adapter, and Hydra config groups exposed through the lerna plugin entry point.
It is storage-focused. Dataset schemas, domain-specific semantics, private bucket conventions, and application storage conventions are not part of this package.
Current Status
- Implemented:
S3Config,S3Credentials,S3ClientCredentials,S3Provider, flexibleS3Sessionwith direct AWS fields orccflow-etlkey/secret credentials,S3Client, client configs for AWS, Backblaze B2, Hetzner Object Storage, Cloudflare R2, and custom S3-compatible stores, shared/credentials/s3/...configs, anonymous unsigned access, S3 operation contexts/results, object reads, binary/text/json/CSV/gzip decoding throughccflow-etlPayloadCodec, pyarrow-backed parquet hooks, object existence checks throughS3ExistsContext, metadata reads throughS3HeadContext, prefix listing throughS3ListContext, paged prefix walks throughS3PrefixWalkContext, object copies throughS3CopyContext, explicit deletes throughS3DeleteContext, additiveS3WriteDataContextwrites for byte/string/dict/list payloads, codec-backed CSV/gzip/parquet write encoding, read-write orchestration throughS3ReadWriteContext, temp-key copy finalization for atomic-style publishes, JSON manifest writes throughS3ObjectManifest,S3ArtifactStoreforccflow-etlartifact write/publish contracts with object identity (head) and version-pinned file materialization (pinned reads needs3:GetObjectVersionon AWS; unversioned objects are checked for replacement after download),fsspec_urlwith fsspec (s3fs) and Polars storage options that reach the same endpoint with the same credentials, for lazy reads with other tools, S3 cache adapter,cache=s3, andoutput=/outputs/s3config groups, and local fake-S3 tests with botocore error semantics. - Partial: write mode skips existing objects unless
overwrite=True, atomic publishes leave the temporary object in place rather than deleting it by default, and parquet support depends onpyarrowthroughccflow-etl. - Missing: manifest readers, temporary-object cleanup policy, live provider smoke tests, and broader moto/botocore-stub integration coverage.
Client And Credentials Config
Select a client group alongside output=/outputs/s3 or cache=s3:
cc-etl +client=/clients/s3/cloudflare +output=/outputs/s3 output.bucket=my-bucket output.prefix=raw
cc-etl +client=/clients/s3/backblaze +cache=s3 cache.store.bucket=my-bucket cache.store.prefix=cache
Available client groups:
| Group | Resolution |
|---|---|
client=/clients/s3/aws |
AWS S3 SDK defaults, optional AWS_REGION or AWS_DEFAULT_REGION. |
client=/clients/s3/backblaze |
BACKBLAZE_S3_ENDPOINT_URL, or https://s3.${BACKBLAZE_S3_REGION}.backblazeb2.com. |
client=/clients/s3/hetzner |
HETZNER_S3_ENDPOINT_URL, or https://${HETZNER_S3_REGION}.your-objectstorage.com. |
client=/clients/s3/cloudflare |
CLOUDFLARE_R2_ENDPOINT_URL, or https://${CLOUDFLARE_R2_ACCOUNT_ID}.r2.cloudflarestorage.com with region auto. |
client=/clients/s3/custom |
S3_ENDPOINT_URL and optional S3_REGION. |
Available standalone credentials groups:
| Group | Resolution |
|---|---|
credentials=/credentials/s3/default |
Boto3 default credential chain. |
credentials=/credentials/s3/aws-profile |
AWS_PROFILE plus optional AWS_REGION. |
credentials=/credentials/s3/aws |
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, optional AWS_SESSION_TOKEN. |
credentials=/credentials/s3/backblaze |
BACKBLAZE_ACCESS_KEY_ID, BACKBLAZE_SECRET_ACCESS_KEY. |
credentials=/credentials/s3/hetzner |
HETZNER_S3_ACCESS_KEY_ID, HETZNER_S3_SECRET_ACCESS_KEY. |
credentials=/credentials/s3/cloudflare |
CLOUDFLARE_R2_ACCESS_KEY_ID, CLOUDFLARE_R2_SECRET_ACCESS_KEY. |
credentials=/credentials/s3/custom |
S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, optional S3_SESSION_TOKEN. |
credentials=/credentials/s3/anonymous |
Unsigned public object access. |
Metadata
Release files for ccflow-s3 0.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ccflow_s3-0.1.2.tar.gz | 23.1 kB | Without provenance | Details for ccflow_s3-0.1.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ccflow_s3-0.1.2-py3-none-any.whl | Python 3 | none | any | Details for ccflow_s3-0.1.2-py3-none-any.whl |
Total release size: 51.7 kB
Release files / ccflow_s3-0.1.2.tar.gz
| Download URL | ccflow_s3-0.1.2.tar.gz |
|---|---|
| Size | 23.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
627bac55556dada171f8db34d1b74fccc01bd49828477a0de210b21f1ac89460
|
|
BLAKE2b-256 checksum How to use checksums |
e7bbdb2acd0ccbd8b9e420ff163a0759c54940ebfe2c2bbbbf49334cc379a0ef
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.15
|
Release files / ccflow_s3-0.1.2-py3-none-any.whl
| Download URL | ccflow_s3-0.1.2-py3-none-any.whl |
|---|---|
| Size | 28.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f8a7fb47e576e28ccdf93e081a9e4c65770d27f277a35239e236c5ab183121c0
|
|
BLAKE2b-256 checksum How to use checksums |
edbe1c166c4cd402ea5ca8b632f47340c7c568b8f2c2b946a7511f9ec5ee4120
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.15
|
Security for ccflow-s3 0.1.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.