acdc-aws-etl-pipeline
Infrastructure and code for the ACDC ETL pipeline and data operations in AWS
Operating the pipeline
Day-to-day operations run through the acdc CLI. Install dependencies, then
explore the command surface and its built-in overview:
poetry install
poetry run acdc --help # all command groups
poetry run acdc docs # mental model, runbook, and prerequisites
New to the project? Start with the CLI Operations Manual — it is the front door for running and inheriting this pipeline (mental model, runbooks, EC2 long jobs, safety rails, and the AWS Glue jobs that live outside the CLI). For an exhaustive listing of every command and flag with examples, see the CLI Command Reference.
Documentation
Core Configuration & Management
- Deployment Configuration Guide
- Dictionary Deployment
- Service Management
- Kubernetes Utilities
- Troubleshooting
Data Lifecycle & ETL
- Data Ingestion
- Data Validation
- Data Transformation (dbt)
- Data Releases
- Data Deletion
- Metadata Deletion by GUID
Metadata & Registry Operations
Analysis & Querying
Library and source code (src/acdc_aws_etl_pipeline)
The Python package in src/acdc_aws_etl_pipeline provides reusable utilities for ingestion, validation, uploads, and Athena/Glue operations used across the pipeline and services.
Modules
ingest/: ingestion helpers for loading source datasets into S3/Glue (seeingest/ingest.py).upload/: Gen3/Sheepdog metadata submission and deletion utilities (e.g.upload/metadata_submitter.py).validate/: schema validation utilities and helpers for validation workflows (seevalidate/validate.py).utils/: shared Athena/Glue/dbt/release helpers (e.g.utils/athena_utils.py,utils/release_writer.py).
Local development
To install dependencies and run tests:
pip install poetry
poetry install
source $(poetry env info --path)/bin/activate
poetry run pytest
Install from PyPI
Releases are published automatically, so you can also install the package directly:
pip install acdc_aws_etl_pipeline
2 maintainers
Metadata
2 maintainers
Release files for acdc_aws_etl_pipeline 1.5.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| acdc_aws_etl_pipeline-1.5.4.tar.gz | 65.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| acdc_aws_etl_pipeline-1.5.4-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 145.1 kB
Release files / acdc_aws_etl_pipeline-1.5.4.tar.gz
| Download URL | acdc_aws_etl_pipeline-1.5.4.tar.gz |
|---|---|
| Size | 65.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
241790c5a32c4b524c190b8ddbc0ef7fa07165168e1a0caf0b51aef214f2cd38
|
|
BLAKE2b-256 checksum How to use checksums |
aef015a40f679d60235183e79dd59c07ec888c07dead0d2d3b260630a3f3cb69
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.2.1 CPython/3.9.25 Linux/6.17.0-1020-azure
|
Release files / acdc_aws_etl_pipeline-1.5.4-py3-none-any.whl
| Download URL | acdc_aws_etl_pipeline-1.5.4-py3-none-any.whl |
|---|---|
| Size | 80.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
aaacd35dc0166f090d5e699d2fe03c2dad8ac7c855114935c91984ec21ef7762
|
|
BLAKE2b-256 checksum How to use checksums |
827a54f5d69dac96eeac7198bf91bfbbad0bf88c1b500c34a22fa1025e0b73d3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.2.1 CPython/3.9.25 Linux/6.17.0-1020-azure
|
