Skip to main content

ETL for Python

Project description

ETL using python

Python package for extracting data from a source, transforming it and loading it to a destination, with validation in between.

The provided ETL pipeline provides useful functionality on top of the usual operations:

  • Extract: Extract data from multiples sources, in parallel (using threads).
  • Validate: Validate the extracted data, to make sure it matches what will be required by the transform step, using pandera schemas. This provide early fail if there is any unexpected change in the sources.
  • Transform: Define the logic for transformation of the data, making it reusable, and allowing multiple data frames as input and multiple data frames as output.
  • Validate again: Validate the transformed data, to make sure it matches your expectation, and what the destination will require.
  • Load: Load multiple data, each to one or more destination, and load diferent data to diferent destinations in parallel (using threads).

Installation

The package is available at PyPI, so you can install it using pip:

pip install extralo

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

extralo-0.17.4.tar.gz (12.1 kB view details)

Uploaded Source

Built Distribution

extralo-0.17.4-py3-none-any.whl (13.9 kB view details)

Uploaded Python 3

File details

Details for the file extralo-0.17.4.tar.gz.

File metadata

  • Download URL: extralo-0.17.4.tar.gz
  • Upload date:
  • Size: 12.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.1 CPython/3.12.7

File hashes

Hashes for extralo-0.17.4.tar.gz
Algorithm Hash digest
SHA256 6cd360bbb6252ef1cadadeabeb194c5a85c3a343c728d794fc91af4b2fc750c5
MD5 26d128ad71bbb8c6f11b2a0159a23c69
BLAKE2b-256 ac638e7b86426b2a0cc9a21901bc3c513850fcfe78ff5ed9bc22f26b0554166b

See more details on using hashes here.

Provenance

The following attestation bundles were made for extralo-0.17.4.tar.gz:

Publisher: ci-cd.yml on vitorcapdeville/extralo

Attestations:

File details

Details for the file extralo-0.17.4-py3-none-any.whl.

File metadata

  • Download URL: extralo-0.17.4-py3-none-any.whl
  • Upload date:
  • Size: 13.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.1 CPython/3.12.7

File hashes

Hashes for extralo-0.17.4-py3-none-any.whl
Algorithm Hash digest
SHA256 5b9416d5b2c68dc8d35973e338f85baa50e1bf06b979e124e6e355b354121940
MD5 ad8964372cda62da795525bf8517b6a4
BLAKE2b-256 5e62107b26909accd1649f1cf89d80857da13060747ff581b0e3c002b5856da2

See more details on using hashes here.

Provenance

The following attestation bundles were made for extralo-0.17.4-py3-none-any.whl:

Publisher: ci-cd.yml on vitorcapdeville/extralo

Attestations:

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page