acdc_aws_etl_pipeline 0.7.4

Tools for ACDC ETL pipeline

acdc-aws-etl-pipeline

Infrastructure and code for the ACDC ETL pipeline and data operations in AWS

Documentation

• Dictionary deployment
• Data ingestion
• Data validation
• Data transformation (dbt)
• Data releases
• Synthetic data generation
• REST API upload to sheepdog
• Data deletion
• IndexD file registration
• Querying Athena
• Writing Athena queries to JSON
• Troubleshooting

Library and source code (src/acdc_aws_etl_pipeline)

The Python package in src/acdc_aws_etl_pipeline provides reusable utilities for ingestion, validation, uploads, and Athena/Glue operations used across the pipeline and services.

Modules

• ingest/: ingestion helpers for loading source datasets into S3/Glue (see ingest/ingest.py).
• upload/: Gen3/Sheepdog metadata submission and deletion utilities (e.g. upload/metadata_submitter.py).
• validate/: schema validation utilities and helpers for validation workflows (see validate/validate.py).
• utils/: shared Athena/Glue/dbt/release helpers (e.g. utils/athena_utils.py, utils/release_writer.py).

Local development

To install dependencies and run tests:

pip install poetry
poetry install
source $(poetry env info --path)/bin/activate
poetry run pytest

Install from PyPI

Releases are published automatically, so you can also install the package directly:

pip install acdc_aws_etl_pipeline