Skip to main content

dlt is an open-source python-first scalable data loading library that does not require any backend to run.

Project description

data load tool (dlt) — the open-source Python library for data loading

Be it a Google Colab notebook, AWS Lambda function, an Airflow DAG, your local laptop,
or a GPT-4 assisted development playground—dlt can be dropped in anywhere.

🚀 Join our thriving community of likeminded developers and build the future together!

Installation

dlt supports Python 3.9 to Python 3.13.

pip install dlt

More options: Install via Conda or Pixi

Quick Start

Load chess game data from chess.com API and save it in DuckDB:

import dlt
from dlt.sources.helpers import requests

# Create a dlt pipeline that will load
# chess player data to the DuckDB destination
pipeline = dlt.pipeline(
    pipeline_name='chess_pipeline',
    destination='duckdb',
    dataset_name='player_data'
)

# Grab some player data from Chess.com API
data = []
for player in ['magnuscarlsen', 'rpragchess']:
    response = requests.get(f'https://api.chess.com/pub/player/{player}')
    response.raise_for_status()
    data.append(response.json())

# Extract, normalize, and load the data
pipeline.run(data, table_name='player')

Try it out in our Colab Demo

Features

  • Automatic Schema: Data structure inspection and schema creation for the destination.
  • Data Normalization: Consistent and verified data before loading.
  • Seamless Integration: Colab, AWS Lambda, Airflow, and local environments.
  • Scalable: Adapts to growing data needs in production.
  • Easy Maintenance: Clear data pipeline structure for updates.
  • Rapid Exploration: Quickly explore and gain insights from new data sources.
  • Versatile Usage: Suitable for ad-hoc exploration to advanced loading infrastructures.
  • Start in Seconds with CLI: Powerful CLI for managing, deploying and inspecting local pipelines.
  • Incremental Loading: Load only new or changed data and avoid loading old records again.
  • Open Source: Free and Apache 2.0 Licensed.

Ready to use Sources and Destinations

Explore ready to use sources (e.g. Google Sheets) in the Verified Sources docs and supported destinations (e.g. DuckDB) in the Destinations docs.

Documentation

For detailed usage and configuration, please refer to the official documentation.

Examples

You can find examples for various use cases in the examples folder.

Adding as dependency

dlt follows the semantic versioning with the MAJOR.MINOR.PATCH pattern.

  • major means breaking changes and removed deprecations
  • minor new features, sometimes automatic migrations
  • patch bug fixes

We suggest that you allow only patch level updates automatically:

Get Involved

The dlt project is quickly growing, and we're excited to have you join our community! Here's how you can get involved:

  • Connect with the Community: Join other dlt users and contributors on our Slack
  • Report issues and suggest features: Please use the GitHub Issues to report bugs or suggest new features. Before creating a new issue, make sure to search the tracker for possible duplicates and add a comment if you find one.
  • Track progress of our work and our plans: Please check out our public Github project
  • Improve documentation: Help us enhance the dlt documentation.

Contribute code

Please read CONTRIBUTING before you make a PR.

  • 📣 New destinations are unlikely to be merged due to high maintenance cost (but we are happy to improve SQLAlchemy destination to handle more dialects)
  • Significant changes require tests and docs and in many cases writing tests will be more laborious than writing code
  • Bugfixes and improvements are welcome! You'll get help with writing tests and docs + a decent review.

License

dlt is released under the Apache 2.0 License.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dlt-1.14.1.tar.gz (776.3 kB view details)

Uploaded Source

Built Distribution

dlt-1.14.1-py3-none-any.whl (992.7 kB view details)

Uploaded Python 3

File details

Details for the file dlt-1.14.1.tar.gz.

File metadata

  • Download URL: dlt-1.14.1.tar.gz
  • Upload date:
  • Size: 776.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: uv/0.6.9

File hashes

Hashes for dlt-1.14.1.tar.gz
Algorithm Hash digest
SHA256 2fb1f65fd9d755e0fd7f8b694655d8e7dd438b5251e8513d0be3fc9c952aae6e
MD5 26be89ea82dc023f95dcece89a28ec09
BLAKE2b-256 1e4f5836de7f12eb9a64442210649b52f885bd115d2afc9f930219653480ce7d

See more details on using hashes here.

File details

Details for the file dlt-1.14.1-py3-none-any.whl.

File metadata

  • Download URL: dlt-1.14.1-py3-none-any.whl
  • Upload date:
  • Size: 992.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: uv/0.6.9

File hashes

Hashes for dlt-1.14.1-py3-none-any.whl
Algorithm Hash digest
SHA256 209b74213684e779ab3d63dcd25e4ab5d125e9f7a5949dae020665e2fc9f6baf
MD5 e46a2b0333d49806019c0f9abfc5a9fd
BLAKE2b-256 bd25d8e2150c31cf21c434d4fd4d14fe0028f2361173413338aedd323161986b

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page