Skip to main content

ccflow-s3

ccflow models for S3

Build Status codecov License PyPI

Overview

ccflow-s3 provides public, domain-neutral S3 and S3-compatible storage callable models for ccflow workflows. It provides client and credential configuration, object reads/writes, existence checks, listing, manifests, atomic publish behavior, an S3-backed cache adapter, a generic artifact store adapter, and Hydra config groups exposed through the lerna plugin entry point.

It is storage-focused. Dataset schemas, domain-specific semantics, private bucket conventions, and application storage conventions are not part of this package.

Current Status

  • Implemented: S3Config, S3Credentials, S3ClientCredentials, S3Provider, flexible S3Session with direct AWS fields or ccflow-etl key/secret credentials, S3Client, client configs for AWS, Backblaze B2, Hetzner Object Storage, Cloudflare R2, and custom S3-compatible stores, shared /credentials/s3/... configs, anonymous unsigned access, S3 operation contexts/results, object reads, binary/text/json/CSV/gzip decoding through ccflow-etl PayloadCodec, pyarrow-backed parquet hooks, object existence checks through S3ExistsContext, metadata reads through S3HeadContext, prefix listing through S3ListContext, paged prefix walks through S3PrefixWalkContext, object copies through S3CopyContext, explicit deletes through S3DeleteContext, additive S3WriteDataContext writes for byte/string/dict/list payloads, codec-backed CSV/gzip/parquet write encoding, read-write orchestration through S3ReadWriteContext, temp-key copy finalization for atomic-style publishes, JSON manifest writes through S3ObjectManifest, S3ArtifactStore for ccflow-etl artifact write/publish contracts with object identity (head) and version-pinned file materialization (pinned reads need s3:GetObjectVersion on AWS; unversioned objects are checked for replacement after download), fsspec_url with fsspec (s3fs) and Polars storage options that reach the same endpoint with the same credentials, for lazy reads with other tools, S3 cache adapter, cache=s3, and output=/outputs/s3 config groups, and local fake-S3 tests with botocore error semantics.
  • Partial: write mode skips existing objects unless overwrite=True, atomic publishes leave the temporary object in place rather than deleting it by default, and parquet support depends on pyarrow through ccflow-etl.
  • Missing: manifest readers, temporary-object cleanup policy, live provider smoke tests, and broader moto/botocore-stub integration coverage.

Client And Credentials Config

Select a client group alongside output=/outputs/s3 or cache=s3:

cc-etl +client=/clients/s3/cloudflare +output=/outputs/s3 output.bucket=my-bucket output.prefix=raw
cc-etl +client=/clients/s3/backblaze +cache=s3 cache.store.bucket=my-bucket cache.store.prefix=cache

Available client groups:

Group Resolution
client=/clients/s3/aws AWS S3 SDK defaults, optional AWS_REGION or AWS_DEFAULT_REGION.
client=/clients/s3/backblaze BACKBLAZE_S3_ENDPOINT_URL, or https://s3.${BACKBLAZE_S3_REGION}.backblazeb2.com.
client=/clients/s3/hetzner HETZNER_S3_ENDPOINT_URL, or https://${HETZNER_S3_REGION}.your-objectstorage.com.
client=/clients/s3/cloudflare CLOUDFLARE_R2_ENDPOINT_URL, or https://${CLOUDFLARE_R2_ACCOUNT_ID}.r2.cloudflarestorage.com with region auto.
client=/clients/s3/custom S3_ENDPOINT_URL and optional S3_REGION.

Available standalone credentials groups:

Group Resolution
credentials=/credentials/s3/default Boto3 default credential chain.
credentials=/credentials/s3/aws-profile AWS_PROFILE plus optional AWS_REGION.
credentials=/credentials/s3/aws AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, optional AWS_SESSION_TOKEN.
credentials=/credentials/s3/backblaze BACKBLAZE_ACCESS_KEY_ID, BACKBLAZE_SECRET_ACCESS_KEY.
credentials=/credentials/s3/hetzner HETZNER_S3_ACCESS_KEY_ID, HETZNER_S3_SECRET_ACCESS_KEY.
credentials=/credentials/s3/cloudflare CLOUDFLARE_R2_ACCESS_KEY_ID, CLOUDFLARE_R2_SECRET_ACCESS_KEY.
credentials=/credentials/s3/custom S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, optional S3_SESSION_TOKEN.
credentials=/credentials/s3/anonymous Unsigned public object access.

Metadata

Release files for ccflow-s3 0.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ccflow-s3 0.1.2
File Size Uploaded Provenance Actions
ccflow_s3-0.1.2.tar.gz 23.1 kB Without provenance Details for ccflow_s3-0.1.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ccflow-s3 0.1.2
File Interpreter ABI Platform Actions
ccflow_s3-0.1.2-py3-none-any.whl Python 3 none any Details for ccflow_s3-0.1.2-py3-none-any.whl

Total release size: 51.7 kB

Release files / ccflow_s3-0.1.2.tar.gz

Download URL ccflow_s3-0.1.2.tar.gz
Size 23.1 kB
Tags Source
SHA-256 checksum
How to use checksums
627bac55556dada171f8db34d1b74fccc01bd49828477a0de210b21f1ac89460
BLAKE2b-256 checksum
How to use checksums
e7bbdb2acd0ccbd8b9e420ff163a0759c54940ebfe2c2bbbbf49334cc379a0ef
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Release files / ccflow_s3-0.1.2-py3-none-any.whl

Download URL ccflow_s3-0.1.2-py3-none-any.whl
Size 28.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f8a7fb47e576e28ccdf93e081a9e4c65770d27f277a35239e236c5ab183121c0
BLAKE2b-256 checksum
How to use checksums
edbe1c166c4cd402ea5ca8b632f47340c7c568b8f2c2b946a7511f9ec5ee4120
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Security for ccflow-s3 0.1.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page