ccflow-etl 0.2.0

ETL Tools for ccflow

ccflow-etl

Domain-neutral ETL building blocks for ccflow callable models.

ccflow-etl provides reusable support primitives for ETL-style workflows built as concrete ccflow CallableModel graphs. It keeps generic execution concerns here and leaves workflow-specific behavior to the package or application that owns the workflow.

Install

pip install ccflow-etl

Connector-backed cache and checkpoint stores are provided by connector packages that own their I/O:

Package	Type	Integration
ccflow-s3	generic, cache	S3-backed cache and checkpoint store
ccflow-db	generic, cache	database-backed cache and checkpoint store
ccflow-email	generic, publisher	email publishers for ETL notifications
ccflow-celery	generic, evaluator	Celery-based evaluator for ETL task execution

Quick Start

ccflow-etl installs shared Hydra entry points for running and explaining configured callables:

cc-etl +context.path=./example-output.json +context.payload.message='hello from ccflow-etl' +context.overwrite=true
cc-etl-explain +context.path=./example-output.json

Most projects provide their own config directory and still use the shared entry point:

cc-etl --config-path ./config --config-name text_stats +context.input_path=./notes.txt +context.output_path=./stats.json

What It Provides

• Shared CLI entry points: cc-etl and cc-etl-explain.
• Date expansion: Interval, BaseCalendar, built-in calendars, BackfillContext, and BackfillModel.
• Generic credential models and a /credentials Hydra registry for package extension.
• Handoff metadata: ETLArtifact for typed stage artifacts.
• Format-aware writes and cache handoffs: LocalWriteModel, CachePutModel, CacheGetModel, PayloadCodec, LocalCacheStore, and no-op cache defaults.
• Checkpointing: CheckpointRecord, checkpoint statuses, CheckpointDecisionModel for idempotent skip decisions, and no-op checkpoint defaults.
• Retry integration: compatibility exports for ccflow RetryPolicy and RetryModel; use ccflow.evaluators.RetryEvaluator for runtime evaluator retries.
• Execution policy: ExecutionPolicy for shared max-concurrency hints and rate spacing that evaluators and connector models can consume through the /execution Hydra group.
• Run reporting: RunSummary for structured counts by status and artifact stage.

Documentation

• CLI And Config
• Building Pipelines
• Backfills And Calendars
• Handoffs, Formats, And Reliability
• API Reference
• Development

Package Boundaries

ccflow-etl owns domain-neutral ETL contracts, generic credential shapes, and helpers. It does not own application workflows, provider clients, connector clients, provider-specific credential semantics, or domain-specific rules. Durable store implementations should live in connector packages and integrate through the generic cache and checkpoint contracts.