ETLantic
Design once. Validate everywhere.
Typed, contract-driven data pipelines for Python.
Documentation · Quickstart · Capabilities · Roadmap
ETLantic is a typed control layer for data pipelines. Define data, transformations, and topology as Python contracts; validate them before work begins; then run or compile the same logical pipeline for different backends.
Typed contracts ──▶ Validation ──▶ Deterministic plan ──▶ Run or compile
The name describes the model: ETL is the data flow; ETLantic surrounds it with typed contracts, validation, planning, and evidence.
V(model) → Extract → V(input) → Transform → V(output) → Load → V/evidence
Validation is a control layer, not another business transformation. Runtime checks are selected by policy and backend capability; publication evidence does not imply an automatic sink reread.
Why ETLantic?
- Catch invalid wiring, incompatible contracts, missing capabilities, and untrusted plugins before a write.
- Validate extracted inputs, transformation outputs, engine transitions, and publication boundaries against the same contracts.
- Keep one logical pipeline across local Python, Polars, Pandas, SQL, PySpark, Airflow, and Prefect.
- Review deterministic, secret-free plans and preserve structured diagnostics, lineage, schema observations, and run reports.
- Install a small core and add only the engines you need.
ETLantic does not replace dataframe engines, databases, Spark, schedulers, storage systems, catalogs, or secret managers. It gives them one typed pipeline model and one inspectable validation lifecycle.
Status: ETLantic 0.23.0 ships runtime resilience and performance budgets on top of the 0.22 Plugin SDK RC: measured microbenchmark envelopes, fault injection, durable-store hardening, and expanded backend CI. Stable for documented single-tenant reference deployments—not unrestricted enterprise production. Structured Streaming remains experimental. See Capabilities and Production readiness.
Quickstart
ETLantic requires Python 3.11 or newer.
pip install etlantic==0.23.0
etlantic --version
mkdir my-pipeline && cd my-pipeline
etlantic init --with-toml
etlantic doctor --profile development
etlantic validate pipeline.py:SamplePipeline --profile development
etlantic plan pipeline.py:SamplePipeline --profile development
etlantic run pipeline.py:SamplePipeline --profile development
cat data/out.json
You should see run status succeeded and JSON rows for Ada and Grace. The CLI
defaults to development when --profile is omitted (or your project's
default_profile). Prefer an explicit profile in scripts and CI.
Full walkthrough: Quickstart.
Next: First Pipeline
(evolve the generated project). From a checkout, an optional in-memory SDK demo
is examples/memory_customers.py.
Engines and integrations
| Integration | Install | Role |
|---|---|---|
| Polars | etlantic-polars |
Eager/lazy dataframe execution and portable compilation |
| Pandas | etlantic-pandas |
Eager dataframe execution and portable compilation |
| SQL | etlantic-sql |
Parameterized relational execution and portable SQL compilation |
| PySpark | etlantic-pyspark |
Spark execution and portable compilation |
| Airflow | etlantic-airflow |
Compile plans into DAG artifacts |
| Prefect | etlantic-prefect |
Direct-execution scheduler integration |
| Keyring | etlantic-keyring |
OS keyring secret provider |
| SQLModel | etlantic-sqlmodel |
SQLModel bridge helpers |
| SparkForge | etlantic-sparkforge |
Medallion adapter (bronze/silver/gold stay out of core) |
| DataFusion | etlantic-datafusion |
Experimental query engine stub (Gate B) |
See Optional packages
for observability (otel / observability extras) and Arrow helpers.
Matching extras such as etlantic[polars] are equivalent. Pin matching minors
while ETLantic is pre-1.0. Airflow is compile-only and does not install Apache
Airflow itself.
Architecture
ETLantic keeps logical meaning separate from physical execution:
Data (ODCS/ContractModel) + Transformation (DTCS) + Pipeline (DPCS)
│
validate and plan
▼
secret-free PipelinePlan
│
┌───────────┼───────────┐
▼ ▼ ▼
execute compile generate
│ │ │
└──── plugins and external systems
Plans and reports contain secret references, never resolved secret values. Production profiles require explicit plugin allowlists. Backend optimizations may change the physical graph but must preserve contracts, validation boundaries, security domains, and logical attribution.
Capability boundary
| Capability | 0.23 |
|---|---|
Cohesive CLI (init, doctor, durable reports) |
Available |
| Typed contracts, graph validation, deterministic planning | Available |
| Local, Polars, Pandas, SQL, and PySpark execution paths | Available |
| Portable compilers for Polars, Pandas, SQL, and PySpark | Available |
| ODCS, DTCS, DPCS, schema drift, lineage, reports, and SARIF | Available |
| Airflow compilation and Prefect scheduling | Available |
| Versioned Polars↔Pandas tabular interchange | Available |
| Contract and configuration freeze (deep plans, security_mode) | Available |
| Trust, isolation, safe I/O, SBOM/attestations | Available |
| Structured Streaming | Experimental |
etlantic-datafusion |
Experimental |
| Full multi-tenant control plane | Not included |
See the full Capabilities and Validation Everywhere guides for precise guarantees and limitations.
Learn more
Installation · Quickstart · Engine selection · Compare · Security · Roadmap · Contributing
MIT licensed.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file etlantic-0.23.0.tar.gz.
File metadata
- Download URL: etlantic-0.23.0.tar.gz
- Upload date:
- Size: 482.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.11.3 {"installer":{"name":"uv","version":"0.11.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
c6c731f4004cca26ac11eb351760dd3550375603560ee6312a27037cdb099082
|
|
| MD5 |
c366f2adab81b448f9088ab280d791df
|
|
| BLAKE2b-256 |
2bfd6395d0e29dcce5599dc369fac3d1a3629ff7343552769877dc272ba79728
|
File details
Details for the file etlantic-0.23.0-py3-none-any.whl.
File metadata
- Download URL: etlantic-0.23.0-py3-none-any.whl
- Upload date:
- Size: 374.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.11.3 {"installer":{"name":"uv","version":"0.11.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
c8519eb03b480bdc0fdbaa950b85055c3050d00e20dcc9df7a3b31375e897587
|
|
| MD5 |
f5eeeaa39510cc6d1a351ae632b9a5d9
|
|
| BLAKE2b-256 |
e5cc67799f0ce776477b0f86bb469cae560b7971784eaadd6429d4469140b89f
|
