Viadot
Documentation: https://viadot.docs.dyvenia.com
Source Code: https://github.com/dyvenia/viadot/tree/main
A simple data ingestion library to guide data flows from some places to other places.
Getting Data from a Source
Viadot supports several API and RDBMS sources, private and public. Currently, we support the UK Carbon Intensity public API and base the examples on it.
from viadot.sources.uk_carbon_intensity import UKCarbonIntensity
ukci = UKCarbonIntensity()
ukci.query("/intensity")
df = ukci.to_df()
print(df)
Output:
| from | to | forecast | actual | index | |
|---|---|---|---|---|---|
| 0 | 2021-08-10T11:00Z | 2021-08-10T11:30Z | 211 | 216 | moderate |
The above df is a pandas DataFrame object. It contains data downloaded by viadot from the Carbon Intensity UK API.
Loading data to a destination
Depending on the destination, viadot provides different methods of uploading data. For instance, for databases, this would be bulk inserts. For data lakes, it would be file uploads.
For example:
from viadot.sources import UKCarbonIntensity
from viadot.sources import AzureDataLake
ukci = UKCarbonIntensity()
ukci.query("/intensity")
df = ukci.to_df()
adls = AzureDataLake(config_key="my_adls_creds")
adls.from_df(df, "my_folder/my_file.parquet")
Getting started
Prerequisites
We use uv. You can install it like so:
curl -LsSf https://astral.sh/uv/install.sh | sh
Installation
pip install viadot2
Running the Docker Environment
The project provides the run.sh script located in the docker/ directory to initialize Docker environment with optional configuration for image tags and target platform.
Make sure the script is executable.
chmod +x docker/run.sh
Usage
docker/run.sh [-t <image_tag>] [-p <platform>]
- -t <image_tag> - set Docker image tag (default: latest)
- -p - set the target platform for Docker (default: linux/amd64)
- This default ensures compatibility on macOS (Apple Silicon). Optionally, you can provide a different platform.
Examples
Run with default settings:
docker/run.sh
Run with a specific image tag and platform:
docker/run.sh -t dev -p linux/amd64
Configuration
In order to start using sources, you must configure them with required credentials. Credentials can be specified either in the viadot config file (by default, $HOME/.config/viadot/config.yaml), or passed directly to each source's credentials parameter.
You can find specific information about each source's credentials in the documentation.
Next steps
Check out the documentation for more information on how to use viadot.
1 maintainer
Metadata
1 maintainer
Release files for viadot2 2.15.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| viadot2-2.15.0.tar.gz | 979.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| viadot2-2.15.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 1.3 MB
Release files / viadot2-2.15.0.tar.gz
| Download URL | viadot2-2.15.0.tar.gz |
|---|---|
| Size | 979.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c66d77ea5e09f0f958e43df215486c8a19df70bd181847eb0d77b14bb6842942
|
|
BLAKE2b-256 checksum How to use checksums |
2d2ccc153570a508ed9660fb4890ab8fd7dc611e153d5f91449fea78bbc3a96a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 23, 2026.
Transparency logRelease files / viadot2-2.15.0-py3-none-any.whl
| Download URL | viadot2-2.15.0-py3-none-any.whl |
|---|---|
| Size | 361.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c6c1f03cef644fd2b4efb34170d14e098715c6210bf7f6cd0b28f2019c8249c4
|
|
BLAKE2b-256 checksum How to use checksums |
150ce555044ba4c606d5c0c05f2d90e8197abfafd56838b89d8f670f1e814529
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 23, 2026.
Transparency log
