ETLantic
Typed Python data pipelines with validate-before-write.
Design once. Validate everywhere.
Documentation · Quickstart · Capabilities · Roadmap
ETLantic lets you define Python data pipelines as typed classes or
functional builders / JSON (PipelineDefinition), catch bad wiring and
contract mismatches before any write, then run or compile the same
pipeline on local Python, Polars, Pandas, SQL, or Spark—and emit Airflow
DAGs when you need them.
It is not a warehouse tool (use dbt), not a scheduler (use Airflow, Dagster, or Prefect), and not a dataframe engine. It is a typed pipeline framework that coordinates the tools you already choose.
Typed contracts ──▶ Validation ──▶ Deterministic plan ──▶ Run or compile
Why ETLantic?
- Catch invalid wiring, incompatible contracts, missing capabilities, and untrusted plugins before a write.
- Validate extracted inputs, transformation outputs, engine transitions, and publication boundaries against the same contracts.
- Keep one logical pipeline across local Python, Polars, Pandas, SQL, and PySpark; compile to Airflow DAGs; run under Prefect where the local MVP applies.
- Review deterministic, secret-free plans and preserve structured diagnostics, lineage, schema observations, and run reports.
- Install a small core and add only the engines you need.
Quickstart
Requires Python 3.11 or newer. Use an empty directory for init (or pass
--force). On Windows PowerShell, activate with
.\.venv\Scripts\Activate.ps1 and prefer py -3.11 -m … (see
Installation).
python -m venv .venv && source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install 'etlantic==0.29.0'
python -m etlantic --version
mkdir my-pipeline && cd my-pipeline
python -m etlantic init --with-toml
python -m etlantic validate pipeline.py:SamplePipeline --profile development
python -m etlantic run pipeline.py:SamplePipeline --profile development
cat data/out.json
You should see run status succeeded and JSON rows for Ada and Grace (identity
transform on the sample). That proves plumbing—next, change the transform in
First Pipeline.
The CLI defaults to development when --profile is omitted (or your project's
default_profile). Prefer an explicit profile in scripts and CI.
Full walkthrough: Quickstart.
After first success (clone only): repository demos under
examples/(for examplememory_customers.py) require a git checkout — they are not in the PyPI wheel.
Status: ETLantic 0.29.0 is a Beta (PyPI) release suitable for documented single-tenant pilots—not unrestricted enterprise production. Structured Streaming remains experimental. See Capabilities and Production readiness.
Engines and integrations
| Integration | Install | Role |
|---|---|---|
| Polars | etlantic-polars |
Eager/lazy dataframe execution and portable compilation |
| Pandas | etlantic-pandas |
Eager dataframe execution and portable compilation |
| SQL | etlantic-sql |
Parameterized relational execution and portable SQL compilation |
| PySpark | etlantic-pyspark |
Spark execution and portable compilation |
| Airflow | etlantic-airflow |
Compile plans into DAG artifacts (does not install Airflow) |
| Prefect | etlantic-prefect |
Direct-execution local MVP (deployment/serve remain future) |
| Keyring | etlantic-keyring |
OS keyring secret provider |
| SQLModel | etlantic-sqlmodel |
SQLModel bridge helpers |
| Medallantic | medallantic |
SparkForge IR → ETLantic migration adapter (native authoring + IR migrate in 0.29; prefer medallantic over deprecated etlantic-sparkforge) |
| DataFusion | etlantic-datafusion |
Experimental query engine stub (Gate B) |
| FastAPI | etlantic-fastapi |
Thin 0.29 authoring/service reference adapter (not the 1.1 control plane) |
See Optional packages
for observability (otel / observability extras) and Arrow helpers.
Matching extras such as etlantic[polars] are equivalent. Pin matching minors
while ETLantic is pre-1.0.
Architecture
ETLantic keeps logical meaning separate from physical execution:
Data + Transformation + Pipeline contracts
│
validate and plan
▼
secret-free PipelinePlan
│
┌───────────┼───────────┐
▼ ▼ ▼
execute compile generate
│ │ │
└──── plugins and external systems
Plans and reports contain secret references, never resolved secret values. Production profiles require explicit plugin allowlists. Backend optimizations may change the physical graph but must preserve contracts, validation boundaries, security domains, and logical attribution.
Contract standards (ODCS / DTCS / DPCS) and the validation envelope are covered in the Architecture and Validation Everywhere guides.
Capability boundary
| Capability | 0.29 |
|---|---|
Cohesive CLI (init, doctor, durable reports) |
Available |
| Typed contracts, graph validation, deterministic planning | Available |
| Local, Polars, Pandas, SQL, and PySpark execution paths | Available |
| Portable compilers for Polars, Pandas, SQL, and PySpark | Available |
| ODCS, DTCS, DPCS, schema drift, lineage, reports, and SARIF | Available |
| Airflow compilation (compile-only) and Prefect local MVP | Available (bounded) |
| Versioned Polars↔Pandas tabular interchange | Available |
| Contract and configuration freeze (deep plans, security_mode) | Available |
| Trust, isolation, safe I/O, SBOM/attestations (single-tenant reference) | Available (bounded) |
| Structured Streaming | Experimental |
etlantic-datafusion |
Experimental |
| Full multi-tenant control plane / SLA / unrestricted enterprise | Not included |
See the full Capabilities guide for precise guarantees and limitations.
Release notes: What's New in 0.29.
Learn more
Installation · Quickstart · Engine selection · Compare · Security · Roadmap · Contributing
MIT licensed.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file etlantic-0.29.0.tar.gz.
File metadata
- Download URL: etlantic-0.29.0.tar.gz
- Upload date:
- Size: 566.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.11.3 {"installer":{"name":"uv","version":"0.11.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
4c4f65173bbc707565ab02c13abb70d4525d156754d98cb1284053d90a8a4c72
|
|
| MD5 |
e14bf96a9708b9736f3df6b8744dcd0c
|
|
| BLAKE2b-256 |
83048b0387c0ae6e2f668372a3568777e35b707ccc216144e1e07fa011458db5
|
File details
Details for the file etlantic-0.29.0-py3-none-any.whl.
File metadata
- Download URL: etlantic-0.29.0-py3-none-any.whl
- Upload date:
- Size: 418.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.11.3 {"installer":{"name":"uv","version":"0.11.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a0cc805d4e90a231580d2a1a4a551cb03fa425261e9f351b399ab7a4861653f8
|
|
| MD5 |
5c46928f3202f7905324e1d791010085
|
|
| BLAKE2b-256 |
b2ef89aa9e7300dde0e17226319f9f35258b2b49621ab51d759bba49c0cfcf25
|
