acdc-aws-etl-pipeline
Infrastructure and code for the ACDC ETL pipeline and data operations in AWS
Documentation
Core Configuration & Management
- Deployment Configuration Guide
- Dictionary Deployment
- Service Management
- Kubernetes Utilities
- Troubleshooting
Data Lifecycle & ETL
- Data Ingestion
- Data Validation
- Data Transformation (dbt)
- Data Releases
- Data Deletion
- Metadata Deletion by GUID
Metadata & Registry Operations
Analysis & Querying
Library and source code (src/acdc_aws_etl_pipeline)
The Python package in src/acdc_aws_etl_pipeline provides reusable utilities for ingestion, validation, uploads, and Athena/Glue operations used across the pipeline and services.
Modules
ingest/: ingestion helpers for loading source datasets into S3/Glue (seeingest/ingest.py).upload/: Gen3/Sheepdog metadata submission and deletion utilities (e.g.upload/metadata_submitter.py).validate/: schema validation utilities and helpers for validation workflows (seevalidate/validate.py).utils/: shared Athena/Glue/dbt/release helpers (e.g.utils/athena_utils.py,utils/release_writer.py).
Local development
To install dependencies and run tests:
pip install poetry
poetry install
source $(poetry env info --path)/bin/activate
poetry run pytest
Install from PyPI
Releases are published automatically, so you can also install the package directly:
pip install acdc_aws_etl_pipeline
Metadata
Release files for acdc_aws_etl_pipeline 1.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| acdc_aws_etl_pipeline-1.1.0.tar.gz | 41.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| acdc_aws_etl_pipeline-1.1.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 88.7 kB
Release files / acdc_aws_etl_pipeline-1.1.0.tar.gz
| Download URL | acdc_aws_etl_pipeline-1.1.0.tar.gz |
|---|---|
| Size | 41.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
31af08715cad1f3a39e34c8f513951ccebf742dbf2888aea372b076073baa994
|
|
BLAKE2b-256 checksum How to use checksums |
acf563e42266281a1ca8205202bb52c228a711aa0c08af9797385e43b4f1862d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.2.1 CPython/3.9.25 Linux/6.17.0-1013-azure
|
Release files / acdc_aws_etl_pipeline-1.1.0-py3-none-any.whl
| Download URL | acdc_aws_etl_pipeline-1.1.0-py3-none-any.whl |
|---|---|
| Size | 47.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
aea88648dc63a167e8312061b88c682e0093b2b3110395910335df6bb528b926
|
|
BLAKE2b-256 checksum How to use checksums |
708eac1d6b37e11cf959e07e0733a0d8eef013958dfcc46be58ed6bc802529b9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.2.1 CPython/3.9.25 Linux/6.17.0-1013-azure
|