dlt-saga
Config-driven data ingestion and historization framework, built on dlt.
Why dlt-saga?
dlt is an excellent Python library for building data pipelines. dlt-saga adds the operational layer that teams need to run dlt at scale:
| What you get | How |
|---|---|
| Zero-code pipelines | Drop a YAML file in configs/ — no Python needed for common sources |
| SCD2 historization | write_disposition: append+historize turns any snapshot table into a full change history with _dlt_valid_from / _dlt_valid_to |
| Docs & classification in the warehouse | Declare description and classification (e.g. [pii]) on tables/columns; saga writes and reconciles them onto the destination (persist_docs) |
| dbt-style selectors | saga ingest --select "tag:daily,group:api" — union, intersection, glob patterns |
| Multi-environment profiles | profiles.yml with dev/prod targets, service account impersonation, per-environment datasets |
| Plugin architecture | Register custom sources and destinations via packages.yml or Python entry points — no framework fork needed |
| Cloud-agnostic | BigQuery today, Databricks and DuckDB included, more via plugins |
If you are already using dlt directly and finding yourself re-implementing incremental state management, environment switching, or SCD2 transforms — dlt-saga is the config layer you are building.
Installation
pip install dlt-saga[gcp] # BigQuery (GCP)
pip install dlt-saga[databricks,azure] # Databricks on Azure
pip install dlt-saga # DuckDB only (no cloud dependencies)
Quick Start
# 1. Create and scaffold a project
mkdir my-pipelines && cd my-pipelines
saga init # prompts for destination and credentials
# 2. Authenticate to your destination (skip for DuckDB)
# See: https://github.com/Glitni/dlt-saga/wiki/Getting-Started
# 3. List available pipelines
saga list
# 4. Run a pipeline
saga ingest --select "example__sample"
See the Getting Started guide for a full walkthrough, or browse
example/for a minimal runnable setup.
Local execution is the default. Use
--orchestrateto fan out to parallel workers (requiresorchestration:configured insaga_project.yml).
CLI Commands
All commands are subcommands under the saga entry point and share common options:
--select, --verbose, --profile, --target.
Selectors (dbt-style)
Selectors filter which pipelines to run. They work across all commands.
| Syntax | Meaning | Example |
|---|---|---|
name |
Exact pipeline name | --select google_sheets__my_pipeline |
*glob* |
Glob pattern | --select "*balance*" |
tag:name |
Filter by tag | --select "tag:daily" (schedule-aware — see Configuration → Scheduling tags) |
group:name |
Filter by source group | --select "group:google_sheets" |
state:new |
Target table doesn't exist yet | --select "state:new" (see CLI Reference → State selectors) |
state:failed |
Last recorded run failed | --select "state:failed" |
| space-separated | UNION (OR) | --select "tag:daily group:filesystem" |
| comma-separated | INTERSECTION (AND) | --select "tag:daily,group:google_sheets" |
Common Examples
# List pipelines
saga list # All enabled pipelines
saga list --resource-type ingest # Ingest-enabled only
saga list --resource-type historize # Historize-enabled only
saga list --select "tag:daily" # Filtered by tag
# Ingest
saga ingest --select "tag:daily"
saga ingest --select "group:api" --workers 8
saga ingest --full-refresh --select "my_pipeline"
saga ingest --select "group:api" --start-value-override "2026-01-01" # Backfill
# Historize (SCD2)
saga historize --select "tag:daily"
saga historize --full-refresh --select "filesystem__*"
# Run (ingest + historize sequentially)
saga run --select "tag:daily"
# Update BigQuery access controls
saga update-access --select "group:google_sheets"
# Target a specific environment
saga ingest --target prod --select "tag:daily" # production (with impersonation)
Adding a New Pipeline
Create a YAML config file in configs/<source_type>/ — that's it. The framework auto-discovers configs.
Supported source types out of the box: API, Database (PostgreSQL, MySQL, SQL Server, and more via ConnectorX), Filesystem (GCS, SFTP, local), Google Sheets, and SharePoint.
See the Pipeline Types guide for config examples for each source type, and the Configuration reference for all available fields.
Write Dispositions and Historize
The write_disposition field controls what operations are enabled for a pipeline:
| Value | Ingest | Historize | Use Case |
|---|---|---|---|
append |
Yes | No | Raw event/log data |
merge |
Yes | No | Upsert on primary key |
replace |
Yes | No | Full refresh each run |
append+historize |
Yes | Yes | Snapshot → SCD2 |
historize |
No | Yes | External data → SCD2 |
Historize transforms raw snapshot data into SCD2 tables with _dlt_valid_from, _dlt_valid_to, and _dlt_is_deleted columns. See the Historize guide for the full reference.
Community
- GitHub Issues — bug reports and feature requests
- GitHub Discussions — questions, ideas, show & tell
- Contributing guide — how to get involved
- dlt community — dlt Slack / Discord
Further Reading
- Getting Started — Full walkthrough: install, init, first pipeline
- Architecture — Three-layer design, plugin system, execution flow
- Pipeline Types — Config reference for API, Database, Filesystem, Sheets, SharePoint
- Configuration — Hierarchical config, all options reference
- Profiles — Multi-environment setup, service account impersonation
- Historize (SCD2) — Snapshot tables → slowly changing dimensions
- CLI Reference — All commands, flags, and the programmatic API
- Deployment — Orchestration, Cloud Run, worker setup
- Performance — Parallel execution, worker tuning, backfill
- Plugin Development — Custom sources, destinations, hooks
Origin
dlt-saga is derived from an internal data ingestion framework originally built by Glitni for Amedia, a leading Nordic media group, as the ingestion layer of Amedia's data platform. Amedia supported open-sourcing the project and continues to fund ongoing development through their partnership with Glitni, enabling the framework to be shared with the broader community.
Project Structure
dlt-saga/
├── dlt_saga/ # Main package
│ ├── cli.py # CLI entry point (saga command)
│ ├── pipelines/ # Built-in source implementations
│ │ ├── api/ # Generic REST API pipeline
│ │ ├── database/ # Database source (ConnectorX)
│ │ ├── filesystem/ # Filesystem / GCS source
│ │ ├── google_sheets/# Google Sheets source
│ │ └── sharepoint/ # SharePoint source
│ ├── historize/ # SCD2 historization engine
│ ├── destinations/ # Destination implementations
│ │ ├── bigquery/ # BigQuery
│ │ └── duckdb/ # DuckDB (local development)
│ ├── pipeline_config/ # Config discovery and parsing
│ ├── schemas/ # Bundled static schemas (dlt_common.json)
│ └── utility/ # Shared utilities (CLI, naming, orchestration)
├── example/ # Minimal runnable consumer project (DuckDB)
├── wiki/ # Documentation (synced to GitHub wiki)
└── .dlt/ # dlt runtime config overrides
1 maintainer
Release files for dlt-saga 0.33.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| dlt_saga-0.33.1.tar.gz | 482.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| dlt_saga-0.33.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 1.0 MB
Release files / dlt_saga-0.33.1.tar.gz
| Download URL | dlt_saga-0.33.1.tar.gz |
|---|---|
| Size | 482.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
cf60042bcf1fa098856108ddb1b149b708ae45106c243672cc70ed3686f02ff5
|
|
BLAKE2b-256 checksum How to use checksums |
c48c999dc5a3412c62f1a2c88668f00815e5bc42b191b4086c1ae1a1c9a356c4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.13
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 25, 2026.
Transparency logRelease files / dlt_saga-0.33.1-py3-none-any.whl
| Download URL | dlt_saga-0.33.1-py3-none-any.whl |
|---|---|
| Size | 548.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d315593730034c29be4d744ea13d979a08f9f70d2ca3bd7cab80c7b84373020d
|
|
BLAKE2b-256 checksum How to use checksums |
4a5b93ef276c2c917ab658079dde76ae858c6cb4fe33917eb159a053b767f8f1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.13
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 25, 2026.
Transparency log
