omniload

Copy data between any source and any destination.

About

omniload is a polyglot data loader framework based on dlt, the open-source Python library for building data pipelines. dlt does the heavy lifting (schema inference, normalization, and incremental load state); omniload wraps it behind a single CLI and a URI scheme, so you can copy data between any source and any destination without writing pipeline code, either from your shell or the Python API.

omniload provides the same efficient incremental data loading modes inherited by dlt: append, merge, and delete+insert.

Install

Container

For running omniload in a container environment, use the provided OCI images.

docker run --rm ghcr.io/panodata/omniload:latest version

Package

For running or installing the omniload Python package, we are recommending to use uv. ¹²

uvx omniload

Alternatively, if you like to install it account-wide on your system:

uv tool install omniload

Synopsis

CLI

Instruct omniload to read from the PostgreSQL table public.some_data, and write to the BigQuery warehouse table omniload.some_data.

omniload ingest \
    --source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \
    --source-table 'public.some_data' \
    --dest-uri 'bigquery://<your-project-name>?credentials_path=/path/to/service/account.json' \
    --dest-table 'omniload.some_data'

Python API

Using omniload.run_ingest, you can run the same ingestion from your own application. The keyword arguments map one-to-one to the CLI options, and it returns the dlt LoadInfo for the run (or None for a dry run).

from omniload import run_ingest

info = run_ingest(
    source_uri="sqlite:///./source.db",
    dest_uri="duckdb:///./warehouse.duckdb",
    source_table="main.some_table",
    dest_table="public.some_table",
)
print(info)

See the Python API documentation for dry runs, error handling, and passing enum options as strings.

Handbook

Please visit the quickstart documentation, or inspect the list of supported sources and destinations.

Project

Contribute

Contributions are very much welcome. Please visit the sandbox documentation to learn how to spin up a development environment on your workstation and submit patches, or create a ticket to report a bug or propose a feature.

Status

Breaking changes should be expected until a 1.0 release, so version pinning is strongly recommended, especially when using this software as a library. For example:

pip install 'omniload[full]==0.0.42'

License

The project is licensed under the MIT License, see the LICENSE file for details. Some components are licensed under the Apache 2.0 license, see the NOTICE file for details.

Acknowledgements

This project would not have been possible without the amazing work by the authors and contributors to SQLAlchemy, dlt, and ingestr, turtles all the way down. Kudos.

You can install uv using pip install uv, or another method that matches your operating system. ↩
While installing omniload with vanilla pip is possible, it is an order of magnitude slower. ↩

Metadata

Release files for omniload 0.18.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for omniload 0.18.0
File	Size	Uploaded
omniload-0.18.0.tar.gz	392.6 kB	Sep 23, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for omniload 0.18.0
File	Interpreter	ABI	Platform	Reset
omniload-0.18.0-py3-none-any.whl 517.6 kB Sep 23, 2026	Python 3	none	any	Details

Total release size: 910.3 kB

Release files / omniload-0.18.0.tar.gz

Download URL	omniload-0.18.0.tar.gz
Size	392.6 kB
Tags	Source
SHA-256 checksum How to use checksums	`c616bd3b8d4124654e46e92b7f08f30d21349dc14b09e3fc5ffbcf19adc2ec6c`
BLAKE2b-256 checksum How to use checksums	`ff59b8ac2f1c1089698f51cb64d752da64a6e8f06deea3c6aabc068c6b29362c`
Upload date	Sep 23, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.11.11`

Release files / omniload-0.18.0-py3-none-any.whl

Download URL	omniload-0.18.0-py3-none-any.whl
Size	517.6 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`2d7d93c50cc6fa877a0017a62377844b52b719a40654de7d25935d6ac363a846`
BLAKE2b-256 checksum How to use checksums	`61181f0a9cd734bd656857f57247771397227472ecce98436729570a6a693bf7`
Upload date	Sep 23, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.11.11`