Skip to main content

Polars and duckdb based json configured simple ETL pipelines

Project description

endurance_etl

Polars and duckdb based json configured simple ETL pipelines

WARNING: PLACEHOLDER FOR LATER DEVELOPMENT

Introduction

Use endurance_etl to execute some simple ETL pipelines:

import json
from endurance_etl import Tars

CONFIG = "sample.json"

with open(CONFIG) as f:
    print(json.load(f))

# Output:
{
    "SOURCES": [
        {
            "name": "csv_file_source",
            "path": "csv_file_source.csv",
            # ...other_kwargs
        }
    ],
    "TARGETS": [
        {
            "name": "csv_file_target",
            "source": "source/csv_file_source",
            "target": "csv_file_target.csv",
            "transforms": [
                {
                    "function": "lambda df: df + 1"
                }
            ]
            # ...other_kwargs
        }
    ]
}

tars = Tars.from_json(CONFIG)
tars.do()

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

endurance_etl-0.1.0.tar.gz (1.9 kB view details)

Uploaded Source

Built Distribution

endurance_etl-0.1.0-py3-none-any.whl (2.4 kB view details)

Uploaded Python 3

File details

Details for the file endurance_etl-0.1.0.tar.gz.

File metadata

  • Download URL: endurance_etl-0.1.0.tar.gz
  • Upload date:
  • Size: 1.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.4.2 CPython/3.11.3 Darwin/22.4.0

File hashes

Hashes for endurance_etl-0.1.0.tar.gz
Algorithm Hash digest
SHA256 681a724c1f5f094424225939bb5e9f5797b7127572ed85d4da39df57f0149c25
MD5 14a2e49f53d1df869d811e0fd987c9cb
BLAKE2b-256 848704b85bd1921f4d973453b2f7d6d4ff9b34974969288f4e331e157a76f2da

See more details on using hashes here.

File details

Details for the file endurance_etl-0.1.0-py3-none-any.whl.

File metadata

  • Download URL: endurance_etl-0.1.0-py3-none-any.whl
  • Upload date:
  • Size: 2.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.4.2 CPython/3.11.3 Darwin/22.4.0

File hashes

Hashes for endurance_etl-0.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 58dffc4b7fa31c53faae22c670b3407fc7402d20ddcc36dfbf126b1466ac9594
MD5 ce191ee39e29c4deddacbf98179e634d
BLAKE2b-256 e7a4e71ec00d6247a69c6c0abfa7c746f16bd320a2cf88f59e2b237ca99ff21a

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page