acdc-aws-etl-pipeline
Infrastructure and code for the ACDC ETL pipeline and data operations in AWS
Documentation
Core Configuration & Management
- Deployment Configuration Guide
- Dictionary Deployment
- Service Management
- Kubernetes Utilities
- Troubleshooting
Data Lifecycle & ETL
- Data Ingestion
- Data Validation
- Data Transformation (dbt)
- Data Releases
- Data Deletion
- Metadata Deletion by GUID
Metadata & Registry Operations
Analysis & Querying
Library and source code (src/acdc_aws_etl_pipeline)
The Python package in src/acdc_aws_etl_pipeline provides reusable utilities for ingestion, validation, uploads, and Athena/Glue operations used across the pipeline and services.
Modules
ingest/: ingestion helpers for loading source datasets into S3/Glue (seeingest/ingest.py).upload/: Gen3/Sheepdog metadata submission and deletion utilities (e.g.upload/metadata_submitter.py).validate/: schema validation utilities and helpers for validation workflows (seevalidate/validate.py).utils/: shared Athena/Glue/dbt/release helpers (e.g.utils/athena_utils.py,utils/release_writer.py).
Local development
To install dependencies and run tests:
pip install poetry
poetry install
source $(poetry env info --path)/bin/activate
poetry run pytest
Install from PyPI
Releases are published automatically, so you can also install the package directly:
pip install acdc_aws_etl_pipeline
Metadata
Release files for acdc_aws_etl_pipeline 1.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| acdc_aws_etl_pipeline-1.2.0.tar.gz | 41.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| acdc_aws_etl_pipeline-1.2.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 88.7 kB
Release files / acdc_aws_etl_pipeline-1.2.0.tar.gz
| Download URL | acdc_aws_etl_pipeline-1.2.0.tar.gz |
|---|---|
| Size | 41.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
55b775e66b0b24070037214852832d3250d26ebbd96a99f3a4f9408a12e8fe5b
|
|
BLAKE2b-256 checksum How to use checksums |
2d0c240f39af3b55e49c7323627cdc24c677c42db7cc5e364c571473a048bb26
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.2.1 CPython/3.9.25 Linux/6.17.0-1018-azure
|
Release files / acdc_aws_etl_pipeline-1.2.0-py3-none-any.whl
| Download URL | acdc_aws_etl_pipeline-1.2.0-py3-none-any.whl |
|---|---|
| Size | 47.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
74897f1e052ab739589f4421bd29bc618a6dfae6c28bbaf8ef06023785c9e505
|
|
BLAKE2b-256 checksum How to use checksums |
90eedd4302c866f2e1bf0c24ebbbe87dc49ab9bd6e438d93089df8b37266c5aa
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.2.1 CPython/3.9.25 Linux/6.17.0-1018-azure
|