About
omniload is a polyglot data loader framework based on dlt, the open-source Python library for building data pipelines. dlt does the heavy lifting (schema inference, normalization, and incremental load state); omniload wraps it behind a single CLI and a URI scheme, so you can copy data between any source and any destination without writing pipeline code, either from your shell or the Python API.
omniload provides the same efficient incremental data loading modes
inherited by dlt: append, merge, and delete+insert.
Install
Container
For running omniload in a container environment, use the provided OCI
images.
docker run --rm ghcr.io/panodata/omniload:latest version
Package
For running or installing the omniload Python package,
we are recommending to use uv. 12
uvx omniload
Alternatively, if you like to install it account-wide on your system:
uv tool install omniload
Synopsis
CLI
Instruct omniload to read from the PostgreSQL table public.some_data,
and write to the BigQuery warehouse table omniload.some_data.
omniload ingest \
--source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \
--source-table 'public.some_data' \
--dest-uri 'bigquery://<your-project-name>?credentials_path=/path/to/service/account.json' \
--dest-table 'omniload.some_data'
Python API
Using omniload.run_ingest, you can run the same
ingestion from your own application. The keyword arguments map one-to-one to the
CLI options, and it returns the dlt LoadInfo for the run (or None for a dry run).
from omniload import run_ingest
info = run_ingest(
source_uri="sqlite:///./source.db",
dest_uri="duckdb:///./warehouse.duckdb",
source_table="main.some_table",
dest_table="public.some_table",
)
print(info)
See the Python API documentation for dry runs, error handling, and passing enum options as strings.
Handbook
Please visit the quickstart documentation, or inspect the list of supported sources and destinations.
Project
Contribute
Contributions are very much welcome. Please visit the sandbox documentation to learn how to spin up a development environment on your workstation and submit patches, or create a ticket to report a bug or propose a feature.
Status
Breaking changes should be expected until a 1.0 release, so version pinning is strongly recommended, especially when using this software as a library. For example:
pip install 'omniload[full]==0.0.42'
License
The project is licensed under the MIT License, see the LICENSE file for details. Some components are licensed under the Apache 2.0 license, see the NOTICE file for details.
Acknowledgements
This project would not have been possible without the amazing work by the authors and contributors to SQLAlchemy, dlt, and ingestr, turtles all the way down. Kudos.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file omniload-0.11.0.tar.gz.
File metadata
- Download URL: omniload-0.11.0.tar.gz
- Upload date:
- Size: 441.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
8376acd1c8c614949f39246be1b287311517e12469d9cc160ff21dd32f31efc1
|
|
| MD5 |
716808dccb92a5a0d6b1e7ebe4a4852a
|
|
| BLAKE2b-256 |
f31076226f01e966249987020062c47774d0dcb2e1d0bb5dbc7709926e2f6daa
|
Provenance
The following attestation bundles were made for omniload-0.11.0.tar.gz:
Publisher:
release-pypi.yml on panodata/omniload
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
omniload-0.11.0.tar.gz -
Subject digest:
8376acd1c8c614949f39246be1b287311517e12469d9cc160ff21dd32f31efc1 - Sigstore transparency entry: 2545920106
- Sigstore integration time:
-
Permalink:
panodata/omniload@e610fc1353aa14f22becf7c1b471273dd46536d0 -
Branch / Tag:
refs/tags/v0.11.0 - Owner: https://github.com/panodata
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release-pypi.yml@e610fc1353aa14f22becf7c1b471273dd46536d0 -
Trigger Event:
push
-
Statement type:
File details
Details for the file omniload-0.11.0-py3-none-any.whl.
File metadata
- Download URL: omniload-0.11.0-py3-none-any.whl
- Upload date:
- Size: 589.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
faf3615f43aea074b1e2b9c8ce6e292ba12a38058af83856790b2b184e061428
|
|
| MD5 |
f156f0c53bdd39d86cb957beb7589125
|
|
| BLAKE2b-256 |
d118f769ec2fc59babf27a93e5f8796b9ef75cdadc440e08b845bebaa784e932
|
Provenance
The following attestation bundles were made for omniload-0.11.0-py3-none-any.whl:
Publisher:
release-pypi.yml on panodata/omniload
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
omniload-0.11.0-py3-none-any.whl -
Subject digest:
faf3615f43aea074b1e2b9c8ce6e292ba12a38058af83856790b2b184e061428 - Sigstore transparency entry: 2545920242
- Sigstore integration time:
-
Permalink:
panodata/omniload@e610fc1353aa14f22becf7c1b471273dd46536d0 -
Branch / Tag:
refs/tags/v0.11.0 - Owner: https://github.com/panodata
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release-pypi.yml@e610fc1353aa14f22becf7c1b471273dd46536d0 -
Trigger Event:
push
-
Statement type:
