Skip to main content

ccflow-etl

Domain-neutral ETL building blocks for ccflow callable models.

Build Status codecov License PyPI

ccflow-etl provides reusable support primitives for ETL-style workflows built as concrete ccflow CallableModel graphs. It keeps generic execution concerns here and leaves workflow-specific behavior to the package or application that owns the workflow.

Install

pip install ccflow-etl

Connector-backed cache and artifact stores are provided by connector packages that own their I/O. Generic checkpointing belongs in ccflow proper.

Package Type Integration
ccflow-s3 generic, storage S3-backed artifact IO and cache
ccflow-db generic, cache database-backed cache store
ccflow-email generic, publisher email publishers for ETL notifications
ccflow-celery generic, evaluator Celery-based evaluator for ETL task execution

Quick Start

ccflow-etl installs shared Hydra entry points for running and explaining configured callables:

cc-etl +context.path=./example-output.json +context.payload.message='hello from ccflow-etl' +context.overwrite=true
cc-etl-explain +context.path=./example-output.json

Most projects provide their own config directory and still use the shared entry point:

cc-etl --config-path ./config --config-name text_stats +context.input_path=./notes.txt +context.output_path=./stats.json

What It Provides

  • Shared CLI entry points: cc-etl and cc-etl-explain.
  • Date expansion: Interval, BaseCalendar, built-in calendars, BackfillContext, and BackfillModel.
  • Generic credential models and a /credentials Hydra registry for package extension.
  • Generic extract task composition through /tasks, /datasets, and /outputs config selections.
  • Handoff metadata: ETLArtifact for typed stage artifacts.
  • Artifact IO contracts: ArtifactExistsModel, ArtifactWriteModel, ArtifactPublishModel, and NoOpArtifactStore for backend-neutral existence checks, writes, publication, and artifact URIs.
  • File artifacts: ArtifactMaterializeModel and ArtifactWriteFileModel stream artifacts to and from local files. When the store exposes head(), materialization saves the source identity next to the downloaded file and reuses a cached file only when that saved identity still matches the source's version ID (or ETag) and size; otherwise it downloads again.
  • Task and output composition: ExtractTaskModel, LocalFileOutput, FsspecArtifactStore, NoOpArtifactStore, and /tasks / /outputs config selections.
  • Format-aware writes and cache handoffs: LocalWriteModel, CachePutModel, CacheGetModel, PayloadCodec, LocalCacheStore, and no-op cache defaults.
  • Retry integration: compatibility exports for ccflow RetryPolicy and RetryModel; use ccflow.evaluators.RetryEvaluator for runtime evaluator retries.
  • Execution policy: ExecutionPolicy for shared max-concurrency hints and rate spacing that evaluators and connector models can consume through the /execution Hydra group.
  • Datasets: DatasetDefinition declares a date-partitioned dataset (key template, store, optional local copy, format, schema, and calendar of expected partitions) and provides keys, existence and missing-partition checks, explain records, writes of bytes or local files (streamed where the store supports it) with lineage sidecars, local-first reads, and Polars lazy scans over a date range (pip install ccflow-etl[polars]). Scans read Parquet partitions from each store's fsspec URL, local or object storage, after one listing per store. An optional expected Arrow schema, given as an fsspec-data reference or inline, is enforced on writes and lets a range be read as one scan (pip install ccflow-etl[fsspec]). DatasetReadModel exposes a range as a graph node. key_prefix, resolved_store() and resolved_local_store() give a dataset's listing prefix and its stores resolved from the registry; resolve_model and artifact_uri do the same for any registry path and store key.
  • Partition planning: DatasetInventory lists a dataset's store once and reports missing partitions and stale ones, whose lineage sidecar records a source object (uri and etag) that has since changed in one of the declared source datasets; latest_only checks just the last partition, for datasets whose partitions are superseding snapshots, and require_sidecars plans present partitions that lack their lineage sidecar. Each inventory is a callable model on a DateRangeContext that returns a DatasetInventoryResult. DatasetPlanModel declares its inventories as dependencies over the same range and returns a DatasetPlanResult whose rebuild ranges are contexts for producing tasks; it builds nothing, and explain mode reads nothing. Current ETags come from a store's list_objects listing (ccflow-s3 S3ArtifactStore).
  • Run reporting: RunSummary for structured counts by status and artifact stage.

Documentation

Package Boundaries

ccflow-etl owns domain-neutral ETL contracts, generic credential shapes, and helpers. It does not own application workflows, provider clients, connector clients, provider-specific credential semantics, catalogs of which datasets exist, dataset-specific schemas, run reporting evaluators, checkpointing, or domain-specific rules. Durable store implementations should live in connector packages and integrate through generic cache and artifact IO contracts.

Metadata

Release files for ccflow-etl 0.4.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ccflow-etl 0.4.3
File Size Uploaded Provenance Actions
ccflow_etl-0.4.3.tar.gz 50.7 kB Without provenance Details for ccflow_etl-0.4.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ccflow-etl 0.4.3
File Interpreter ABI Platform Actions
ccflow_etl-0.4.3-py3-none-any.whl Python 3 none any Details for ccflow_etl-0.4.3-py3-none-any.whl

Total release size: 118.9 kB

Release files / ccflow_etl-0.4.3.tar.gz

Download URL ccflow_etl-0.4.3.tar.gz
Size 50.7 kB
Tags Source
SHA-256 checksum
How to use checksums
ff9a4d3eb60e1061debf176b4014a1f34e507b91c639d49ca867ca3517ee5abf
BLAKE2b-256 checksum
How to use checksums
c2417c100d5e546f1dba93fa1c11412978cefebaf4c94359a9793201e5e6cae9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Release files / ccflow_etl-0.4.3-py3-none-any.whl

Download URL ccflow_etl-0.4.3-py3-none-any.whl
Size 68.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b37754445596474962aff44ff372e86663c666957c47eb5e5469bbc2dfa27559
BLAKE2b-256 checksum
How to use checksums
6374b695dad36623b8ac3374cad75772c6bdfc5c299c37accf61b9f354372735
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Security for ccflow-etl 0.4.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page