data load tool (dlt) — the open-source Python library that automates all your tedious data loading tasks

Be it a Google Colab notebook, AWS Lambda function, an Airflow DAG, your local laptop,
or an AI coding agent—dlt can be dropped in anywhere.

🚀 Join our thriving community of likeminded developers and build the future together!

Installation

dlt supports Python 3.10 through Python 3.14. Note that some optional extras are not yet available for Python 3.14, so support for this version is considered experimental.

pip install dlt

Add the extras you need for your sources and destinations, for example:

pip install "dlt[duckdb]"        # local DuckDB destination
pip install "dlt[bigquery]"      # or snowflake, postgres, redshift, databricks, athena, ...
pip install "dlt[s3]"            # or gs, az for cloud filesystems
pip install "dlt[sql_database]"  # read from any SQL database
pip install "dlt[hub]"           # data quality, transformations, and AI (see below)

Prefer uv? uv add "dlt[duckdb]".

Quick Start

Describe an API declaratively and load it into DuckDB — dlt handles requests, pagination, schema inference, and typing for you:

import dlt
from dlt.sources.rest_api import rest_api_source

# 1. Describe the API declaratively
source = rest_api_source({
    "client": {"base_url": "https://pokeapi.co/api/v2/"},
    "resources": [
        {"name": "pokemon", "endpoint": {"path": "pokemon", "params": {"limit": 1000}}},
    ],
})

# 2. Point a pipeline at any destination
pipeline = dlt.pipeline(
    pipeline_name="pokemon",
    destination="duckdb",
    dataset_name="pokemon_data",
)

# 3. Extract, normalize, and load
print(pipeline.run(source))

# 4. ...and read it straight back as a DataFrame
print(pipeline.dataset().pokemon.df())

...or load any Python iterable — a resource is just a generator, and dlt infers the schema, types the columns, and writes the table:

import dlt

@dlt.resource(table_name="players", primary_key="id", write_disposition="merge")
def players():
    yield {"id": 1, "name": "Magnus", "rating": 2839}
    yield {"id": 2, "name": "Pragg", "rating": 2758}

dlt.pipeline(destination="duckdb", dataset_name="chess").run(players())

Check out a super simple demo in Colab or a more advanced Hugging Face demo with Marimo notebooks.

Why dlt

dlt loads data from messy, often unstructured sources into well-structured, typed datasets. It's a library, not a platform — you pip install it into your existing code and keep your workflow and the other tools you already use. No black boxes: clean Pythonic interfaces, human-readable file formats, schemas you can inspect, no hidden side effects.

dlt and its docs are built from the ground up for LLMs and coding agents. Pair the typed, declarative primitives below with dlthub.com/context and the LLM-native workflow to go from prompt to working pipeline — across 5000+ sources — often in a single shot.

Extract from any source

REST APIs — describe the endpoints declaratively; filter, map, and flatten records right at the source (docs):

from dlt.sources.rest_api import rest_api_source

source = rest_api_source({
    "client": {
        "base_url": "https://api.example.com/v1",
        "paginator": {"type": "cursor", "cursor_path": "next_cursor"},
    },
    "resources": [
        {
            "name": "guests",
            "endpoint": {"path": "events/guests"},
            "processing_steps": [
                {"filter": lambda r: r["approval_status"] == "approved"},
                {"map": lambda r: {**r, "email": r["email"].lower()}},
            ],
        },
    ],
})

SQL databases — reflect tables and types straight from the database (docs):

from dlt.sources.sql_database import sql_database

source = sql_database("mysql+pymysql://user:pass@host/db")

Files in any bucket — list, then parse CSV / JSONL / Parquet from local disk, S3, GCS, or Azure (docs):

from dlt.sources.filesystem import filesystem, read_csv_duckdb

source = (
    filesystem(bucket_url="s3://my-bucket/data", file_glob="*.csv")
    | read_csv_duckdb()
).with_name("events")

DataFrames & Arrow — pandas, Polars, and Arrow tables load directly; Arrow-backed frames move with zero copies:

import dlt
import pandas as pd

df = pd.DataFrame({"event": ["dlt summit", "DuckCon"], "signups": [1240, 860]})
dlt.pipeline(destination="duckdb", dataset_name="events").run(df, table_name="events")

See many more sources in the ecosystem.

Load to 20+ destinations — swap one string

The same resource runs anywhere. Change the destination string and dlt takes care of credentials, DDL in the target dialect, staging, and schema drift:

pipeline = dlt.pipeline(
    pipeline_name="luma",
    destination="duckdb",       # → snowflake, bigquery, postgres, redshift, databricks,
    dataset_name="luma_data",   #   athena, clickhouse, motherduck, filesystem (S3/GCS/Azure),
)                               #   iceberg, delta, ... and custom reverse-ETL destinations
pipeline.run(source)

dlt handles the parts you'd rather not:

Credentials → secrets.toml / env vars, injected automatically
DDL → CREATE TABLE in the target's dialect
Type mapping → source types converted to the destination's types
Staging → S3 / GCS for warehouses that need it
Schema drift → ALTER TABLE on the fly

Browse all supported destinations, or build a custom one.

Declare intent with decorators

Decorators let you declare what you want — incremental loading, merge strategies, schema contracts, column hints — instead of hand-rolling it. Every knob can be overridden at runtime (docs):

import dlt

@dlt.resource(
    primary_key="id",
    write_disposition="merge",                       # upsert on the primary key
    columns={"email": {"x-annotation-pii": True}},   # type and annotate columns
    schema_contract={"columns": "freeze"},           # reject unexpected columns
)
def events(
    updated_at=dlt.sources.incremental("updated_at"),  # load only new/changed rows
):
    yield from fetch_events(since=updated_at.last_value)


@dlt.source
def luma(api_key: str = dlt.secrets.value):
    return events(), guests()   # group one or more resources behind shared config/auth

Schema contracts enforce the shape at the gate, with three modes — evolve (accept and adapt the schema), freeze (reject the record), and discard (drop the offending row/column) — applied independently to tables, columns, and data_type. You also get schema inference, normalization of nested data, incremental loading, and secrets & config injection out of the box.

Read your data back: the Dataset API

A pipeline is durable. Reconnect to one by name with dlt.attach and read any table back in the shape that fits your tool (docs):

import dlt

pipeline = dlt.attach(pipeline_name="luma", destination="duckdb", dataset_name="luma_data")

dataset = pipeline.dataset()
dataset.tables               # ['events', 'guests', ...]

guests = dataset.guests      # a lazy dlt.Relation
guests.df()                  # pandas DataFrame
guests.arrow()               # pyarrow.Table (zero-copy)
guests.to_ibis()             # ibis expression — lazy, composable

Transform with Ibis — Python in, SQL out

Lift any loaded table into an Ibis expression, compose group-bys, joins, and window functions in Python, and let dlt compile it to SQL in the destination's dialect. Nothing runs until you ask for the result:

import ibis

guests = pipeline.dataset().guests.to_ibis()

guests_by_event = (
    guests
    .group_by("event_id")
    .aggregate(n_guests=ibis._.api_id.count())
)

guests_by_event.to_pyarrow()   # compiles to SQL and runs on the destination

dlt also supports Python and SQL data access, transformations, pipeline inspection, and visualizing data in Marimo notebooks.

Documentation

For detailed usage and configuration, please refer to the official documentation.

Examples

You can find examples for various use cases in the examples folder, or in the code examples section of our docs page.

Adding as dependency

dlt follows the semantic versioning with the MAJOR.MINOR.PATCH pattern.

major means breaking changes and removed deprecations
minor new features, sometimes automatic migrations
patch bug fixes

We suggest that you allow only patch level updates automatically using the Compatible Release Specifier. For example dlt~=1.23.0 allows only versions >=1.23.0 and less than <1.24.0

Please also see our release notes for notable changes between versions.

Get Involved

The dlt project is quickly growing, and we're excited to have you join our community! Here's how you can get involved:

Connect with the Community: Join other dlt users and contributors on our Slack
Report issues and suggest features: Please use the GitHub Issues to report bugs or suggest new features. Before creating a new issue, make sure to search the tracker for possible duplicates and add a comment if you find one.
Track progress of our work and our plans: Please check out our public Github project
Improve documentation: Help us enhance the dlt documentation.

Contribute code

Please read CONTRIBUTING before you make a PR.

📣 New destinations are unlikely to be merged due to high maintenance cost (but we are happy to improve SQLAlchemy destination to handle more dialects)
Significant changes require tests and docs and in many cases writing tests will be more laborious than writing code
Bugfixes and improvements are welcome! You'll get help with writing tests and docs + a decent review.

License

dlt is released under the Apache 2.0 License.

athena az bigquery cli clickhouse databricks dbml deltalake dremio duckdb ducklake fabric filesystem gcp gs hf http hub lance lancedb motherduck mssql oracle parquet polars postgis postgres pyiceberg qdrant redshift s3 sftp snowflake sql-database sqlalchemy synapse weaviate

Release files for dlt 1.30.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dlt 1.30.0
File	Size	Uploaded
dlt-1.30.0.tar.gz	1.2 MB	Aug 11, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for dlt 1.30.0
File	Interpreter	ABI	Platform	Reset
dlt-1.30.0-py3-none-any.whl 1.4 MB Aug 11, 2026	Python 3	none	any	Details

Total release size: 2.6 MB

Release files / dlt-1.30.0.tar.gz

Download URL	dlt-1.30.0.tar.gz
Size	1.2 MB
Tags	Source
SHA-256 checksum How to use checksums	`46157b4c75aabde40c8b12af005e27d51ddde693ebbc2d338682ee0b19527d5b`
BLAKE2b-256 checksum How to use checksums	`93a8fab4e86b8c9a6f7c04c5ecdb9a7d18297b6ecf6c92e13115eed033714ee7`
Upload date	Aug 11, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.11.9 {"installer":{"name":"uv","version":"0.11.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}`

Release files / dlt-1.30.0-py3-none-any.whl

Download URL	dlt-1.30.0-py3-none-any.whl
Size	1.4 MB
Tags	Python 3
SHA-256 checksum How to use checksums	`7e3c66fc9f8874438539e15123c7ff4f587b5779939e5fca3a43bb3e865cbdab`
BLAKE2b-256 checksum How to use checksums	`c7097111a1dfda0b1a92648854182507df1a0c53b17cd258ea5dd206bc65d11f`
Upload date	Aug 11, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.11.9 {"installer":{"name":"uv","version":"0.11.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}`