acdc_aws_etl_pipeline 0.8.0

Tools for ACDC ETL pipeline

acdc-aws-etl-pipeline

Infrastructure and code for the ACDC ETL pipeline and data operations in AWS

Documentation

Core Configuration & Management

• Deployment Configuration Guide
• Dictionary Deployment
• Service Management
• Kubernetes Utilities
• Troubleshooting

Data Lifecycle & ETL

• Data Ingestion
• Data Validation
• Data Transformation (dbt)
• Data Releases
• Data Deletion

Metadata & Registry Operations

• REST API Upload to Sheepdog
• IndexD File Registration
• Synthetic Data Generation

Analysis & Querying

• Querying Athena
• Writing Athena Queries to JSON

Library and source code (src/acdc_aws_etl_pipeline)

The Python package in src/acdc_aws_etl_pipeline provides reusable utilities for ingestion, validation, uploads, and Athena/Glue operations used across the pipeline and services.

Modules

• ingest/: ingestion helpers for loading source datasets into S3/Glue (see ingest/ingest.py).
• upload/: Gen3/Sheepdog metadata submission and deletion utilities (e.g. upload/metadata_submitter.py).
• validate/: schema validation utilities and helpers for validation workflows (see validate/validate.py).
• utils/: shared Athena/Glue/dbt/release helpers (e.g. utils/athena_utils.py, utils/release_writer.py).

Local development

To install dependencies and run tests:

pip install poetry
poetry install
source $(poetry env info --path)/bin/activate
poetry run pytest

Install from PyPI

Releases are published automatically, so you can also install the package directly:

pip install acdc_aws_etl_pipeline