Viadot

UV formatting

Documentation: https://viadot.docs.dyvenia.com

Source Code: https://github.com/dyvenia/viadot/tree/main

A simple data ingestion library to guide data flows from some places to other places.

Getting Data from a Source

Viadot supports several API and RDBMS sources, private and public. Currently, we support the UK Carbon Intensity public API and base the examples on it.

from viadot.sources.uk_carbon_intensity import UKCarbonIntensity

ukci = UKCarbonIntensity()
ukci.query("/intensity")
df = ukci.to_df()

print(df)

Output:

from to forecast actual index
0 2021-08-10T11:00Z 2021-08-10T11:30Z 211 216 moderate

The above df is a pandas DataFrame object. It contains data downloaded by viadot from the Carbon Intensity UK API.

Loading data to a destination

Depending on the destination, viadot provides different methods of uploading data. For instance, for databases, this would be bulk inserts. For data lakes, it would be file uploads.

For example:

from viadot.sources import UKCarbonIntensity
from viadot.sources import AzureDataLake

ukci = UKCarbonIntensity()
ukci.query("/intensity")
df = ukci.to_df()

adls = AzureDataLake(config_key="my_adls_creds")
adls.from_df(df, "my_folder/my_file.parquet")

Getting started

Prerequisites

We use uv. You can install it like so:

curl -LsSf https://astral.sh/uv/install.sh | sh

Installation

pip install viadot2

Running the Docker Environment

The project provides the run.sh script located in the docker/ directory to initialize Docker environment with optional configuration for image tags and target platform.

Make sure the script is executable.

chmod +x docker/run.sh

Usage

docker/run.sh [-t <image_tag>] [-p <platform>]
  • -t <image_tag> - set Docker image tag (default: latest)
  • -p - set the target platform for Docker (default: linux/amd64)
    • This default ensures compatibility on macOS (Apple Silicon). Optionally, you can provide a different platform.

Examples

Run with default settings:

docker/run.sh

Run with a specific image tag and platform:

docker/run.sh -t dev -p linux/amd64

Configuration

In order to start using sources, you must configure them with required credentials. Credentials can be specified either in the viadot config file (by default, $HOME/.config/viadot/config.yaml), or passed directly to each source's credentials parameter.

You can find specific information about each source's credentials in the documentation.

Next steps

Check out the documentation for more information on how to use viadot.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (2.13.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for dyvenia from gravatar.com dyvenia

Credits

Author: acivitillo

Requires

Python >=3.10

Provides Extra

aws azure databricks sap

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

viadot2-2.12.0.tar.gz (964.5 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

viadot2-2.12.0-py3-none-any.whl (346.3 kB view details)

Uploaded Python 3

File details

Details for the file viadot2-2.12.0.tar.gz.

File metadata

  • Download URL: viadot2-2.12.0.tar.gz
  • Upload date:
  • Size: 964.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for viadot2-2.12.0.tar.gz
Algorithm Hash digest
SHA256 f9c2fbc32f4946da76dfdc1a2032c04d974b3675dc98cfb4201d7c1746ff80eb
MD5 46ef9e621b74861979b3a7a46a58e87b
BLAKE2b-256 d99205f583ccd6d711b54b2c575e8f18c50bae7bbd6353cd75feaae37d6719a8

See more details on using hashes here.

Provenance

The following attestation bundles were made for viadot2-2.12.0.tar.gz:

Publisher: cd.yml on dyvenia/viadot

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file viadot2-2.12.0-py3-none-any.whl.

File metadata

  • Download URL: viadot2-2.12.0-py3-none-any.whl
  • Upload date:
  • Size: 346.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for viadot2-2.12.0-py3-none-any.whl
Algorithm Hash digest
SHA256 dd82a7557c0fdec13fac056e11f1645864ecf58c62ee28e4eba448bb4cc2d063
MD5 5817801f903a64d5c90c2fdc93a734d6
BLAKE2b-256 76daf0cc7f848f3ea8b237112c4610ff9840246b6293dbd613edacf9a5a5a957

See more details on using hashes here.

Provenance

The following attestation bundles were made for viadot2-2.12.0-py3-none-any.whl:

Publisher: cd.yml on dyvenia/viadot

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

2.13.0

2 files

This release

2.12.0 This release

2 files

2.11.1

2 files

2.11.0

2 files

2.10.2

2 files

2.10.1

2 files

2.10.0

2 files

2.9.4

2 files

2.9.3

2 files

2.9.2

2 files

2.9.1

2 files

2.8.4

2 files

2.8.3

2 files

2.8.2

2 files

2.8.1

2 files

2.8.0

2 files

2.7.2

2 files

2.7.1

2 files

2.7.0

2 files

2.6.1

2 files

2.6.0

2 files

2.5.4

2 files

2.5.3

2 files

2.5.2

2 files

2.5.1

2 files

2.4.9

2 files

2.4.8

2 files

2.4.7

2 files

2.4.6

2 files

2.4.5

2 files

2.4.4

2 files

2.4.3

2 files

2.4.2

2 files

2.4.1

2 files

2.4.0

2 files

2.3.5

2 files

Pre-release

2.3.5rc1

2 files

2.3.4

2 files

2.3.3

2 files

2.3.2

2 files

2.3.1

2 files

2.3.0

2 files

2.2.20

2 files

2.2.19

2 files

2.2.18

2 files

2.2.17

2 files

2.2.16

2 files

2.2.15

2 files

2.2.14

2 files

2.2.13

2 files

2.2.12

2 files

2.2.11

2 files

2.2.10

2 files

2.2.9

2 files

2.2.8

2 files

2.2.7

2 files

2.2.6

2 files

2.2.5

2 files

2.2.4

2 files

2.2.3

2 files

2.2.2

2 files

2.2.1

2 files

2.2.0

2 files

2.1.30

2 files

2.1.29

2 files

2.1.28

2 files

2.1.27

2 files

2.1.26

2 files

2.1.25

2 files

2.1.24

2 files

2.1.23

2 files

2.1.22

2 files

2.1.21

2 files

2.1.20

2 files

2.1.19

2 files

2.1.18

2 files

2.1.17

2 files

2.1.16

2 files

2.1.15

2 files

2.1.14

2 files

2.1.13

2 files

2.1.12

2 files

Pre-release

2.0a26

2 files

Pre-release

2.0a25

2 files

Pre-release

2.0a24

2 files

Pre-release

2.0a23

2 files

Pre-release

2.0a22

2 files

Pre-release

2.0a21

2 files

Pre-release

2.0a20

2 files

Pre-release

2.0a19

2 files

Pre-release

2.0a18

2 files

Pre-release

2.0a17

2 files

Pre-release

2.0a16

2 files

Pre-release

2.0a15

2 files

Pre-release

2.0a14

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page