Skip to main content

ccflow-etl

Domain-neutral ETL building blocks for ccflow callable models.

Build Status codecov License PyPI

ccflow-etl provides reusable support primitives for ETL-style workflows built as concrete ccflow CallableModel graphs. It keeps generic execution concerns here and leaves workflow-specific behavior to the package or application that owns the workflow.

Install

pip install ccflow-etl

Connector-backed cache and artifact stores are provided by connector packages that own their I/O. Generic checkpointing belongs in ccflow proper.

Package Type Integration
ccflow-s3 generic, storage S3-backed artifact IO and cache
ccflow-db generic, cache database-backed cache store
ccflow-email generic, publisher email publishers for ETL notifications
ccflow-celery generic, evaluator Celery-based evaluator for ETL task execution

Quick Start

ccflow-etl installs shared Hydra entry points for running and explaining configured callables:

cc-etl +context.path=./example-output.json +context.payload.message='hello from ccflow-etl' +context.overwrite=true
cc-etl-explain +context.path=./example-output.json

Most projects provide their own config directory and still use the shared entry point:

cc-etl --config-path ./config --config-name text_stats +context.input_path=./notes.txt +context.output_path=./stats.json

What It Provides

  • Shared CLI entry points: cc-etl and cc-etl-explain.
  • Date expansion: Interval, BaseCalendar, built-in calendars, BackfillContext, and BackfillModel.
  • Generic credential models and a /credentials Hydra registry for package extension.
  • Generic extract task composition through /tasks, /datasets, and /outputs config selections.
  • Handoff metadata: ETLArtifact for typed stage artifacts.
  • Artifact IO contracts: ArtifactExistsModel, ArtifactWriteModel, ArtifactPublishModel, and NoOpArtifactStore for backend-neutral existence checks, writes, publication, and artifact URIs.
  • File artifacts: ArtifactMaterializeModel and ArtifactWriteFileModel stream artifacts to and from local files. When the store exposes head(), materialization saves the source identity next to the downloaded file and reuses a cached file only when that saved identity still matches the source's version ID (or ETag) and size; otherwise it downloads again.
  • Task and output composition: ExtractTaskModel, LocalFileOutput, FsspecArtifactStore, NoOpArtifactStore, and /tasks / /outputs config selections.
  • Format-aware writes and cache handoffs: LocalWriteModel, CachePutModel, CacheGetModel, PayloadCodec, LocalCacheStore, and no-op cache defaults.
  • Retry integration: compatibility exports for ccflow RetryPolicy and RetryModel; use ccflow.evaluators.RetryEvaluator for runtime evaluator retries.
  • Execution policy: ExecutionPolicy for shared max-concurrency hints and rate spacing that evaluators and connector models can consume through the /execution Hydra group.
  • Datasets: DatasetDefinition declares a date-partitioned dataset (key template, store, optional local copy, format, schema, and calendar of expected partitions) and provides keys, existence and missing-partition checks, explain records, writes with lineage sidecars, local-first reads, and Polars lazy scans over a date range (pip install ccflow-etl[polars]). Scans read Parquet partitions from each store's fsspec URL, local or object storage, after one listing per store. An optional expected Arrow schema, given as an fsspec-data reference or inline, is enforced on writes and lets a range be read as one scan (pip install ccflow-etl[fsspec]). DatasetReadModel exposes a range as a graph node.
  • Run reporting: RunSummary for structured counts by status and artifact stage.

Documentation

Package Boundaries

ccflow-etl owns domain-neutral ETL contracts, generic credential shapes, and helpers. It does not own application workflows, provider clients, connector clients, provider-specific credential semantics, dataset inventories, dataset-specific schemas, run reporting evaluators, checkpointing, or domain-specific rules. Durable store implementations should live in connector packages and integrate through generic cache and artifact IO contracts.

Metadata

Release files for ccflow-etl 0.4.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ccflow-etl 0.4.2
File Size Uploaded Provenance Actions
ccflow_etl-0.4.2.tar.gz 44.8 kB Without provenance Details for ccflow_etl-0.4.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ccflow-etl 0.4.2
File Interpreter ABI Platform Actions
ccflow_etl-0.4.2-py3-none-any.whl Python 3 none any Details for ccflow_etl-0.4.2-py3-none-any.whl

Total release size: 106.1 kB

Release files / ccflow_etl-0.4.2.tar.gz

Download URL ccflow_etl-0.4.2.tar.gz
Size 44.8 kB
Tags Source
SHA-256 checksum
How to use checksums
ba71b4e05513cde3c11e97066ba78f1c5cba5c75af432d4e9fd5277df8d43523
BLAKE2b-256 checksum
How to use checksums
1e31b19508d95eccb85bc50d9f22fef233221cbd4df5ef8f26a0025e3c77e2be
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Release files / ccflow_etl-0.4.2-py3-none-any.whl

Download URL ccflow_etl-0.4.2-py3-none-any.whl
Size 61.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
56c5b5dc34f50dec2fd74962757d61d590a42094fbd406bbb29d4634b75e64a5
BLAKE2b-256 checksum
How to use checksums
8799195196138689bce9e8f4612ecaaf3299c3f181a742ae243b1f35e4472c8e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Security for ccflow-etl 0.4.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page