ccflow-etl

Domain-neutral ETL building blocks for ccflow callable models.

ccflow-etl provides reusable support primitives for ETL-style workflows built as concrete ccflow CallableModel graphs. It keeps generic execution concerns here and leaves workflow-specific behavior to the package or application that owns the workflow.

Install

pip install ccflow-etl

Connector-backed cache and artifact stores are provided by connector packages that own their I/O. Generic checkpointing belongs in ccflow proper.

Quick Start

ccflow-etl installs shared Hydra entry points for running and explaining configured callables:

cc-etl +context.path=./example-output.json +context.payload.message='hello from ccflow-etl' +context.overwrite=true
cc-etl-explain +context.path=./example-output.json

Most projects provide their own config directory and still use the shared entry point:

cc-etl --config-path ./config --config-name text_stats +context.input_path=./notes.txt +context.output_path=./stats.json

What It Provides

• Shared CLI entry points: cc-etl and cc-etl-explain.
• Date expansion: Interval, BaseCalendar, built-in calendars, BackfillContext, and BackfillModel.
• Generic credential models and a /credentials Hydra registry for package extension.
• Generic extract task composition through /tasks, /datasets, and /outputs config selections.
• Handoff metadata: ETLArtifact for typed stage artifacts.
• Artifact IO contracts: ArtifactExistsModel, ArtifactWriteModel, ArtifactPublishModel, and NoOpArtifactStore for backend-neutral existence checks, writes, publication, and artifact URIs.
• File artifacts: ArtifactMaterializeModel and ArtifactWriteFileModel stream artifacts to and from local files. When the store exposes head(), materialization saves the source identity next to the downloaded file and reuses a cached file only when that saved identity still matches the source's version ID (or ETag) and size; otherwise it downloads again.
• Task and output composition: ExtractTaskModel, LocalFileOutput, NoOpArtifactStore, and /tasks / /outputs config selections.
• Format-aware writes and cache handoffs: LocalWriteModel, CachePutModel, CacheGetModel, PayloadCodec, LocalCacheStore, and no-op cache defaults.
• Retry integration: compatibility exports for ccflow RetryPolicy and RetryModel; use ccflow.evaluators.RetryEvaluator for runtime evaluator retries.
• Execution policy: ExecutionPolicy for shared max-concurrency hints and rate spacing that evaluators and connector models can consume through the /execution Hydra group.
• Datasets: DatasetDefinition declares a date-partitioned dataset (key template, store, optional local copy, format, schema, and calendar of expected partitions) and provides keys, existence and missing-partition checks, explain records, writes with lineage sidecars, local-first reads, and Polars lazy scans over a date range (pip install ccflow-etl[polars]). DatasetReadModel exposes a range as a graph node.
• Run reporting: RunSummary for structured counts by status and artifact stage.

Documentation

• CLI And Config
• Building Pipelines
• Backfills And Calendars
• Handoffs, Formats, And Reliability
• API Reference
• Development

Package Boundaries

ccflow-etl owns domain-neutral ETL contracts, generic credential shapes, and helpers. It does not own application workflows, provider clients, connector clients, provider-specific credential semantics, dataset inventories, dataset-specific schemas, run reporting evaluators, checkpointing, or domain-specific rules. Durable store implementations should live in connector packages and integrate through generic cache and artifact IO contracts.