ccflow-etl
Domain-neutral ETL building blocks for ccflow callable models.
ccflow-etl provides reusable support primitives for ETL-style workflows built as concrete ccflow CallableModel graphs. It keeps generic execution concerns here and leaves workflow-specific behavior to the package or application that owns the workflow.
Install
pip install ccflow-etl
Connector-backed cache and artifact stores are provided by connector packages that own their I/O. Generic checkpointing belongs in ccflow proper.
| Package | Type | Integration |
|---|---|---|
ccflow-s3 |
generic, storage | S3-backed artifact IO and cache |
ccflow-db |
generic, cache | database-backed cache store |
ccflow-email |
generic, publisher | email publishers for ETL notifications |
ccflow-celery |
generic, evaluator | Celery-based evaluator for ETL task execution |
Quick Start
ccflow-etl installs shared Hydra entry points for running and explaining configured callables:
cc-etl +context.path=./example-output.json +context.payload.message='hello from ccflow-etl' +context.overwrite=true
cc-etl-explain +context.path=./example-output.json
Most projects provide their own config directory and still use the shared entry point:
cc-etl --config-path ./config --config-name text_stats +context.input_path=./notes.txt +context.output_path=./stats.json
What It Provides
- Shared CLI entry points:
cc-etlandcc-etl-explain. - Date expansion:
Interval,BaseCalendar, built-in calendars,BackfillContext, andBackfillModel. - Generic credential models and a
/credentialsHydra registry for package extension. - Generic extract task composition through
/tasks,/datasets, and/outputsconfig selections. - Handoff metadata:
ETLArtifactfor typed stage artifacts. - Artifact IO contracts:
ArtifactExistsModel,ArtifactWriteModel,ArtifactPublishModel, andNoOpArtifactStorefor backend-neutral existence checks, writes, publication, and artifact URIs. - File artifacts:
ArtifactMaterializeModelandArtifactWriteFileModelstream artifacts to and from local files. When the store exposeshead(), materialization saves the source identity next to the downloaded file and reuses a cached file only when that saved identity still matches the source's version ID (or ETag) and size; otherwise it downloads again. - Task and output composition:
ExtractTaskModel,LocalFileOutput,FsspecArtifactStore,NoOpArtifactStore, and/tasks//outputsconfig selections. - Format-aware writes and cache handoffs:
LocalWriteModel,CachePutModel,CacheGetModel,PayloadCodec,LocalCacheStore, and no-op cache defaults. - Retry integration: compatibility exports for
ccflowRetryPolicyandRetryModel; useccflow.evaluators.RetryEvaluatorfor runtime evaluator retries. - Execution policy:
ExecutionPolicyfor shared max-concurrency hints and rate spacing that evaluators and connector models can consume through the/executionHydra group. - Datasets:
DatasetDefinitiondeclares a date-partitioned dataset (key template, store, optional local copy, format, schema, and calendar of expected partitions) and provides keys, existence and missing-partition checks, explain records, writes with lineage sidecars, local-first reads, and Polars lazy scans over a date range (pip install ccflow-etl[polars]). Scans read Parquet partitions from each store's fsspec URL, local or object storage, after one listing per store. An optional expected Arrow schema, given as an fsspec-data reference or inline, is enforced on writes and lets a range be read as one scan (pip install ccflow-etl[fsspec]).DatasetReadModelexposes a range as a graph node. - Run reporting:
RunSummaryfor structured counts by status and artifact stage.
Documentation
- CLI And Config
- Building Pipelines
- Backfills And Calendars
- Handoffs, Formats, And Reliability
- API Reference
- Development
Package Boundaries
ccflow-etl owns domain-neutral ETL contracts, generic credential shapes, and helpers. It does not own application workflows, provider clients, connector clients, provider-specific credential semantics, dataset inventories, dataset-specific schemas, run reporting evaluators, checkpointing, or domain-specific rules. Durable store implementations should live in connector packages and integrate through generic cache and artifact IO contracts.
Metadata
Release files for ccflow-etl 0.4.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ccflow_etl-0.4.2.tar.gz | 44.8 kB | Without provenance | Details for ccflow_etl-0.4.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ccflow_etl-0.4.2-py3-none-any.whl | Python 3 | none | any | Details for ccflow_etl-0.4.2-py3-none-any.whl |
Total release size: 106.1 kB
Release files / ccflow_etl-0.4.2.tar.gz
| Download URL | ccflow_etl-0.4.2.tar.gz |
|---|---|
| Size | 44.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ba71b4e05513cde3c11e97066ba78f1c5cba5c75af432d4e9fd5277df8d43523
|
|
BLAKE2b-256 checksum How to use checksums |
1e31b19508d95eccb85bc50d9f22fef233221cbd4df5ef8f26a0025e3c77e2be
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.15
|
Release files / ccflow_etl-0.4.2-py3-none-any.whl
| Download URL | ccflow_etl-0.4.2-py3-none-any.whl |
|---|---|
| Size | 61.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
56c5b5dc34f50dec2fd74962757d61d590a42094fbd406bbb29d4634b75e64a5
|
|
BLAKE2b-256 checksum How to use checksums |
8799195196138689bce9e8f4612ecaaf3299c3f181a742ae243b1f35e4472c8e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.15
|
Security for ccflow-etl 0.4.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.