acdc-aws-etl-pipeline

Infrastructure and code for the ACDC ETL pipeline and data operations in AWS

Operating the pipeline

Day-to-day operations run through the acdc CLI. Install dependencies, then explore the command surface and its built-in overview:

poetry install
poetry run acdc --help    # all command groups
poetry run acdc docs      # mental model, runbook, and prerequisites

New to the project? Start with the CLI Operations Manual — it is the front door for running and inheriting this pipeline (mental model, runbooks, EC2 long jobs, safety rails, and the AWS Glue jobs that live outside the CLI). For an exhaustive listing of every command and flag with examples, see the CLI Command Reference.

Documentation

Core Configuration & Management

Data Lifecycle & ETL

Metadata & Registry Operations

Analysis & Querying

Library and source code (src/acdc_aws_etl_pipeline)

The Python package in src/acdc_aws_etl_pipeline provides reusable utilities for ingestion, validation, uploads, and Athena/Glue operations used across the pipeline and services.

Modules

Local development

To install dependencies and run tests:

pip install poetry
poetry install
source $(poetry env info --path)/bin/activate
poetry run pytest

Install from PyPI

Releases are published automatically, so you can also install the package directly:

pip install acdc_aws_etl_pipeline

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.5.4)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for harjo391 from gravatar.com harjo391

Credits

Author: JoshuaHarris391

Requires

Python <4.0.0, >=3.9.5

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

acdc_aws_etl_pipeline-1.5.0.tar.gz (60.6 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

acdc_aws_etl_pipeline-1.5.0-py3-none-any.whl (75.3 kB view details)

Uploaded Python 3

File details

Details for the file acdc_aws_etl_pipeline-1.5.0.tar.gz.

File metadata

  • Download URL: acdc_aws_etl_pipeline-1.5.0.tar.gz
  • Upload date:
  • Size: 60.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/2.2.1 CPython/3.9.25 Linux/6.17.0-1020-azure

File hashes

Hashes for acdc_aws_etl_pipeline-1.5.0.tar.gz
Algorithm Hash digest
SHA256 908befce65355aa84f1b87a5b5db055c1866400b888735de0dda67fd1a5dd959
MD5 413f002b5c3b438be643c5717f473e6e
BLAKE2b-256 cb778c0c6896a89286e5127703c44d71b6751c72b11133f181dcfc46d3ff3ac2

See more details on using hashes here.

File details

Details for the file acdc_aws_etl_pipeline-1.5.0-py3-none-any.whl.

File metadata

File hashes

Hashes for acdc_aws_etl_pipeline-1.5.0-py3-none-any.whl
Algorithm Hash digest
SHA256 73adb3cd1f501e0bf50637f59ffc181a521374afff90f2122aebb6ca819ad042
MD5 4ec08572fa0677c1a7cb27bc8892c2e3
BLAKE2b-256 9cbe1f38b08d6f2bdf4a215c0f5a07bfb66a8a70677092192d0febd05ab7461d

See more details on using hashes here.

Release history Release notifications | RSS feed

1.5.4

2 files

1.5.3

2 files

1.5.2

2 files

1.5.1

2 files

This release

1.5.0 This release

2 files

1.2.0

2 files

1.1.0

2 files

1.0.1

2 files

1.0.0

2 files

0.9.9

2 files

0.9.8

2 files

0.9.7

2 files

0.9.6

2 files

0.9.5

2 files

0.9.4

2 files

0.9.3

2 files

0.9.2

2 files

0.9.1

2 files

0.9.0

2 files

0.8.2

2 files

0.8.1

2 files

0.8.0

2 files

0.7.6

2 files

0.7.5

2 files

0.7.4

2 files

0.7.3

2 files

0.7.2

2 files

0.7.1

2 files

0.7.0

2 files

0.6.9

2 files

0.6.8

2 files

0.6.7

2 files

0.6.6

2 files

0.6.5

2 files

0.6.4

2 files

0.6.3

2 files

0.6.2

2 files

0.6.1

2 files

0.6.0

2 files

0.5.9

2 files

0.5.8

2 files

0.5.7

2 files

0.5.6

2 files

0.5.5

2 files

0.5.4

2 files

0.5.3

2 files

0.5.2

2 files

0.5.1

2 files

0.5.0

2 files

0.4.9

2 files

0.4.8

2 files

0.4.7

2 files

0.4.6

2 files

0.4.5

2 files

0.4.3

2 files

0.4.2

2 files

0.4.1

2 files

0.4.0

2 files

0.3.9

2 files

0.3.8

2 files

0.3.7

2 files

0.3.6

2 files

0.3.5

2 files

0.3.4

2 files

0.3.3

2 files

0.3.2

2 files

0.3.1

2 files

0.3.0

2 files

0.2.9

2 files

0.2.8

2 files

0.2.7

2 files

0.2.6

2 files

0.2.5

2 files

0.2.4

2 files

0.2.3

2 files

0.2.2

2 files

0.2.1

2 files

0.2.0

2 files

0.1.9

2 files

0.1.8

2 files

0.1.7

2 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page