ETLantic

One typed pipeline model. Many execution backends.
Typed contracts. Deterministic plans. Pluggable execution.

Quickstart · Documentation · Is ETLantic for me? · Python API · CLI

Incompatible Load wiring fails validate with PMPIPE210 before any successful write (data/out.json stays untouched until you restore Load[Row]). Try the Quickstart aha:

python -m etlantic validate pipeline.py:SamplePipeline --profile development
# → PMPIPE210: … expects Other … received Row …

ETLantic gives Python data pipelines one portable, typed logical model. It coordinates contracts, transformations, and topology without replacing the tools that execute them. Before execution reaches a write, ETLantic checks wiring, contract compatibility, backend capabilities, and plugin trust, then produces a deterministic plan for local engines, backend plugins, or external orchestrators.

It is not a dataframe engine, warehouse transformation system, or scheduler. ETLantic coordinates those tools through one typed logical model.

Python types + pipeline topology
              │
              ▼
      validate before write
              │
              ▼
   deterministic, secret-free plan
              │
       ┌──────┼────────┐
       ▼      ▼        ▼
      run   compile  generate

Five-minute quickstart

ETLantic requires Python 3.11 or newer. Create and activate a virtual environment outside the project that ETLantic will generate.

macOS or Linux:

python -m venv .venv
source .venv/bin/activate

Windows PowerShell:

py -m venv .venv
.\.venv\Scripts\Activate.ps1

Then install ETLantic and initialize a fresh project directory:

python -m pip install --upgrade pip
python -m pip install 'etlantic==0.40.0'
python -m etlantic --version

mkdir my-pipeline
cd my-pipeline
python -m etlantic init --with-toml
python -m etlantic validate pipeline.py:SamplePipeline --profile development
python -m etlantic run pipeline.py:SamplePipeline --profile development

Inspect the result:

cat data/out.json

Or, in PowerShell:

Get-Content data\out.json

You should see a succeeded run and two JSON rows for Ada and Grace.

Continue with the full Quickstart to see ETLantic reject an incompatible contract before a write, then build your first transformation.

Why teams use ETLantic

• Fail before side effects. Catch invalid graph wiring, incompatible contracts, missing engine capabilities, and plugin-trust failures before publication.
• Review what will run. Produce deterministic plans that can be inspected, fingerprinted, diffed, and retained as build evidence.
• Keep contracts at every boundary. Apply the same typed expectations to extracted inputs, transformation outputs, engine transitions, and loads.
• Separate intent from execution. Keep one logical pipeline while plugins own Polars, Pandas, SQL, and PySpark execution or Airflow compilation.
• Adopt incrementally. Start with local Python and JSON files, then install only the engines and integrations you need.
• Automate enforcement. Emit human, JSON, or SARIF diagnostics for local development and CI.

Cross-engine execution is explicit rather than magical: each transformation needs either an implementation for the selected backend or a portable transformation supported by that backend's compiler.

Where ETLantic fits

Read the full comparison guide before adopting ETLantic as a replacement for an existing tool. It is usually a complementary control layer.

Choose an execution path

Core installs without dataframe engines, database drivers, Spark, Airflow, or Prefect. Add only what the pipeline uses. The middle column describes the capability shipped in 0.40, not future roadmap intent.

For controlled deployments, pin core and every official plugin to the same tested release. See engine selection, compatibility, and optional packages.

Structured Streaming and etlantic-datafusion are experimental. The FastAPI package is a thin reference adapter, not a production control plane.

The authoring model

ETLantic pipelines have four public building blocks:

Application code should prefer the curated facade:

import etlantic as etl

Pipelines can be authored as typed classes or with functional builders and versioned PipelineDefinition JSON. Start with the generated project above; then use the SDK tutorial or programmatic authoring guide.

Contracts as build artifacts

Generate a reviewable contract bundle only after the pipeline validates:

python -m etlantic validate pipeline.py:SamplePipeline --format json
python -m etlantic generate pipeline.py:SamplePipeline -o contracts/

contracts/
├── data/              # data contracts
├── transformations/   # transformation contracts
└── pipelines/         # pipeline topology contract

ETLantic integrates with the Open Data Contract Standard (ODCS), Data Transformation Contract Standard (DTCS), and Data Pipeline Contract Standard (DPCS). Generation is deterministic and refuses invalid pipelines.

Security and production posture

ETLantic is currently Beta. It is suitable for documented, controlled, single-tenant pilots—not unrestricted enterprise production.

Use separate processes or stronger infrastructure boundaries for distinct tenants and trust domains. Before a pilot, review:

• Capabilities and limitations
• Production readiness
• Security model
• Enterprise evaluation guide
• Planned multi-tenant control-plane program
• Security reporting policy

Common workflows

# Inspect the logical graph
python -m etlantic inspect pipeline.py:SamplePipeline --format json

# Resolve a deterministic plan
python -m etlantic plan pipeline.py:SamplePipeline \
  --profile development --format json

# Emit SARIF for CI
python -m etlantic validate pipeline.py:SamplePipeline \
  --profile development --format sarif

# Compile through the optional Airflow package
python -m etlantic compile pipeline.py:SamplePipeline \
  --profile development --target airflow -o dags/

# Inspect durable run reports
python -m etlantic report list

The public CLI also includes doctor, profile, diff, plugin, schema, reliability, and viz. See the CLI reference for exit codes and mutation behavior.

Documentation

Repository examples require a clone and are not included in the wheel. Pip users should begin with etlantic init; contributors can use examples/ after uv sync --locked.

Contributing and support

Contributions are welcome. Start with CONTRIBUTING.md for setup, test scopes, documentation checks, and pull-request expectations.

• Usage questions and bug reports: GitHub Issues
• Vulnerabilities: private reporting instructions
• Release history: CHANGELOG.md
• Project governance: GOVERNANCE.md

ETLantic is available under the MIT License.