acdc-aws-etl-pipeline

Infrastructure and code for the ACDC ETL pipeline and data operations in AWS

Operating the pipeline

Day-to-day operations run through the acdc CLI. Install dependencies, then explore the command surface and its built-in overview:

poetry install
poetry run acdc --help    # all command groups
poetry run acdc docs      # mental model, runbook, and prerequisites

New to the project? Start with the CLI Operations Manual — it is the front door for running and inheriting this pipeline (mental model, runbooks, EC2 long jobs, safety rails, and the AWS Glue jobs that live outside the CLI). For an exhaustive listing of every command and flag with examples, see the CLI Command Reference.

Documentation

Core Configuration & Management

Data Lifecycle & ETL

Metadata & Registry Operations

Analysis & Querying

Library and source code (src/acdc_aws_etl_pipeline)

The Python package in src/acdc_aws_etl_pipeline provides reusable utilities for ingestion, validation, uploads, and Athena/Glue operations used across the pipeline and services.

Modules

Local development

To install dependencies and run tests:

pip install poetry
poetry install
source $(poetry env info --path)/bin/activate
poetry run pytest

Install from PyPI

Releases are published automatically, so you can also install the package directly:

pip install acdc_aws_etl_pipeline

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.5.4)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for harjo391 from gravatar.com harjo391

Credits

Author: JoshuaHarris391

Requires

Python <4.0.0, >=3.9.5

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

acdc_aws_etl_pipeline-1.5.1.tar.gz (60.7 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

acdc_aws_etl_pipeline-1.5.1-py3-none-any.whl (75.4 kB view details)

Uploaded Python 3

File details

Details for the file acdc_aws_etl_pipeline-1.5.1.tar.gz.

File metadata

  • Download URL: acdc_aws_etl_pipeline-1.5.1.tar.gz
  • Upload date:
  • Size: 60.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/2.2.1 CPython/3.9.25 Linux/6.17.0-1020-azure

File hashes

Hashes for acdc_aws_etl_pipeline-1.5.1.tar.gz
Algorithm Hash digest
SHA256 b83c81ee62ea044a52407386cc819668c96f84928f9619c2caae0d7b792f5dde
MD5 a3306d4a4c6c701008e234df657cc09b
BLAKE2b-256 96c6410855b768a2151490124127463d9fe70ab01c4f8b486474df6214d4133a

See more details on using hashes here.

File details

Details for the file acdc_aws_etl_pipeline-1.5.1-py3-none-any.whl.

File metadata

File hashes

Hashes for acdc_aws_etl_pipeline-1.5.1-py3-none-any.whl
Algorithm Hash digest
SHA256 210523f2ffcedd89d42902417bd22ce74a139e041e513553585e20ea914161b0
MD5 b04e81437d69551a11ce88a7c9bf38c8
BLAKE2b-256 070022263e6c9e4ba50231e8629786b54c10e2b288add2cd781165ef38c1cbaa

See more details on using hashes here.

Release history Release notifications | RSS feed

1.5.4

2 files

1.5.3

2 files

1.5.2

2 files

This release

1.5.1 This release

2 files

1.5.0

2 files

1.2.0

2 files

1.1.0

2 files

1.0.1

2 files

1.0.0

2 files

0.9.9

2 files

0.9.8

2 files

0.9.7

2 files

0.9.6

2 files

0.9.5

2 files

0.9.4

2 files

0.9.3

2 files

0.9.2

2 files

0.9.1

2 files

0.9.0

2 files

0.8.2

2 files

0.8.1

2 files

0.8.0

2 files

0.7.6

2 files

0.7.5

2 files

0.7.4

2 files

0.7.3

2 files

0.7.2

2 files

0.7.1

2 files

0.7.0

2 files

0.6.9

2 files

0.6.8

2 files

0.6.7

2 files

0.6.6

2 files

0.6.5

2 files

0.6.4

2 files

0.6.3

2 files

0.6.2

2 files

0.6.1

2 files

0.6.0

2 files

0.5.9

2 files

0.5.8

2 files

0.5.7

2 files

0.5.6

2 files

0.5.5

2 files

0.5.4

2 files

0.5.3

2 files

0.5.2

2 files

0.5.1

2 files

0.5.0

2 files

0.4.9

2 files

0.4.8

2 files

0.4.7

2 files

0.4.6

2 files

0.4.5

2 files

0.4.3

2 files

0.4.2

2 files

0.4.1

2 files

0.4.0

2 files

0.3.9

2 files

0.3.8

2 files

0.3.7

2 files

0.3.6

2 files

0.3.5

2 files

0.3.4

2 files

0.3.3

2 files

0.3.2

2 files

0.3.1

2 files

0.3.0

2 files

0.2.9

2 files

0.2.8

2 files

0.2.7

2 files

0.2.6

2 files

0.2.5

2 files

0.2.4

2 files

0.2.3

2 files

0.2.2

2 files

0.2.1

2 files

0.2.0

2 files

0.1.9

2 files

0.1.8

2 files

0.1.7

2 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page