etlantic 0.21.0

Typed, contract-driven data pipeline modeling for Python.

ETLantic

Design once. Validate everywhere.
Typed, contract-driven data pipelines for Python.

Documentation · Quickstart · Capabilities · Roadmap

---

ETLantic is a typed control layer for data pipelines. Define data, transformations, and topology as Python contracts; validate them before work begins; then run or compile the same logical pipeline for different backends.

Typed contracts ──▶ Validation ──▶ Deterministic plan ──▶ Run or compile

The name describes the model: ETL is the data flow; ETLantic surrounds it with typed contracts, validation, planning, and evidence.

V(model) → Extract → V(input) → Transform → V(output) → Load → V/evidence

Validation is a control layer, not another business transformation. Runtime checks are selected by policy and backend capability; publication evidence does not imply an automatic sink reread.

Why ETLantic?

• Catch invalid wiring, incompatible contracts, missing capabilities, and untrusted plugins before a write.
• Validate extracted inputs, transformation outputs, engine transitions, and publication boundaries against the same contracts.
• Keep one logical pipeline across local Python, Polars, Pandas, SQL, PySpark, Airflow, and Prefect.
• Review deterministic, secret-free plans and preserve structured diagnostics, lineage, schema observations, and run reports.
• Install a small core and add only the engines you need.

ETLantic does not replace dataframe engines, databases, Spark, schedulers, storage systems, catalogs, or secret managers. It gives them one typed pipeline model and one inspectable validation lifecycle.

Status: ETLantic 0.21.0 ships a cohesive CLI journey with durable local workspaces. Stable for documented single-tenant reference deployments, not unrestricted enterprise production. Structured Streaming remains experimental. See Capabilities and Production readiness.

Quickstart

ETLantic requires Python 3.11 or newer.

pip install etlantic==0.21.0
etlantic --version

mkdir my-pipeline && cd my-pipeline
etlantic init --with-toml
etlantic doctor --profile development
etlantic validate pipeline.py:SamplePipeline --profile development
etlantic plan pipeline.py:SamplePipeline --profile development
etlantic run pipeline.py:SamplePipeline --profile development
cat data/out.json

You should see run status succeeded and JSON rows for Ada and Grace. The CLI defaults to development when --profile is omitted (or your project's default_profile). Prefer an explicit profile in scripts and CI.

Full walkthrough: Quickstart. Next: First Pipeline (evolve the generated project). From a checkout, an optional in-memory SDK demo is examples/memory_customers.py.

Engines and integrations

Integration	Install	Role
Polars	etlantic-polars	Eager/lazy dataframe execution and portable compilation
Pandas	etlantic-pandas	Eager dataframe execution and portable compilation
SQL	etlantic-sql	Parameterized relational execution and portable SQL compilation
PySpark	etlantic-pyspark	Spark execution and portable compilation
Airflow	etlantic-airflow	Compile plans into DAG artifacts
Prefect	etlantic-prefect	Direct-execution scheduler integration
Keyring	etlantic-keyring	OS keyring secret provider
SQLModel	etlantic-sqlmodel	SQLModel bridge helpers
SparkForge	etlantic-sparkforge	Medallion adapter (bronze/silver/gold stay out of core)
DataFusion	etlantic-datafusion	Experimental query engine stub (Gate B)

See Optional packages for observability (otel / observability extras) and Arrow helpers.

Matching extras such as etlantic[polars] are equivalent. Pin matching minors while ETLantic is pre-1.0. Airflow is compile-only and does not install Apache Airflow itself.

Architecture

ETLantic keeps logical meaning separate from physical execution:

Data (ODCS/ContractModel) + Transformation (DTCS) + Pipeline (DPCS)
                              │
                       validate and plan
                              ▼
                    secret-free PipelinePlan
                              │
                  ┌───────────┼───────────┐
                  ▼           ▼           ▼
               execute      compile     generate
                  │           │           │
                  └──── plugins and external systems

Plans and reports contain secret references, never resolved secret values. Production profiles require explicit plugin allowlists. Backend optimizations may change the physical graph but must preserve contracts, validation boundaries, security domains, and logical attribution.

Capability boundary

Capability	0.21
Cohesive CLI (init, doctor, durable reports)	Available
Typed contracts, graph validation, deterministic planning	Available
Local, Polars, Pandas, SQL, and PySpark execution paths	Available
Portable compilers for Polars, Pandas, SQL, and PySpark	Available
ODCS, DTCS, DPCS, schema drift, lineage, reports, and SARIF	Available
Airflow compilation and Prefect scheduling	Available
Versioned Polars↔Pandas tabular interchange	Available
Contract and configuration freeze (deep plans, security_mode)	Available
Trust, isolation, safe I/O, SBOM/attestations	Available
Structured Streaming	Experimental
etlantic-datafusion	Experimental
Full multi-tenant control plane	Not included

See the full Capabilities and Validation Everywhere guides for precise guarantees and limitations.

Learn more

Installation · Quickstart · Engine selection · Compare · Security · Roadmap · Contributing

MIT licensed.