acdc-aws-etl-pipeline
Infrastructure and code for the ACDC ETL pipeline and data operations in AWS
Operating the pipeline
Day-to-day operations run through the acdc CLI. Install dependencies, then
explore the command surface and its built-in overview:
poetry install
poetry run acdc --help # all command groups
poetry run acdc docs # mental model, runbook, and prerequisites
New to the project? Start with the CLI Operations Manual — it is the front door for running and inheriting this pipeline (mental model, runbooks, EC2 long jobs, safety rails, and the AWS Glue jobs that live outside the CLI). For an exhaustive listing of every command and flag with examples, see the CLI Command Reference.
Documentation
Core Configuration & Management
- Deployment Configuration Guide
- Dictionary Deployment
- Service Management
- Kubernetes Utilities
- Troubleshooting
Data Lifecycle & ETL
- Data Ingestion
- Data Validation
- Data Transformation (dbt)
- Data Releases
- Data Deletion
- Metadata Deletion by GUID
Metadata & Registry Operations
Analysis & Querying
Library and source code (src/acdc_aws_etl_pipeline)
The Python package in src/acdc_aws_etl_pipeline provides reusable utilities for ingestion, validation, uploads, and Athena/Glue operations used across the pipeline and services.
Modules
ingest/: ingestion helpers for loading source datasets into S3/Glue (seeingest/ingest.py).upload/: Gen3/Sheepdog metadata submission and deletion utilities (e.g.upload/metadata_submitter.py).validate/: schema validation utilities and helpers for validation workflows (seevalidate/validate.py).utils/: shared Athena/Glue/dbt/release helpers (e.g.utils/athena_utils.py,utils/release_writer.py).
Local development
To install dependencies and run tests:
pip install poetry
poetry install
source $(poetry env info --path)/bin/activate
poetry run pytest
Install from PyPI
Releases are published automatically, so you can also install the package directly:
pip install acdc_aws_etl_pipeline
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file acdc_aws_etl_pipeline-1.5.3.tar.gz.
File metadata
- Download URL: acdc_aws_etl_pipeline-1.5.3.tar.gz
- Upload date:
- Size: 65.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
poetry/2.2.1 CPython/3.9.25 Linux/6.17.0-1020-azure
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
bb89c7b08f96ea1636fd59b302c7a15a09bdfbc887d8c6e0bdadbe4b2d4b025a
|
|
| MD5 |
d3d84e4d62414f123eee87a04cf007b1
|
|
| BLAKE2b-256 |
611a1c722d5c999aa49996ef79b32700afc8862e72a4a2d301335d1356d35aa6
|
File details
Details for the file acdc_aws_etl_pipeline-1.5.3-py3-none-any.whl.
File metadata
- Download URL: acdc_aws_etl_pipeline-1.5.3-py3-none-any.whl
- Upload date:
- Size: 80.1 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
poetry/2.2.1 CPython/3.9.25 Linux/6.17.0-1020-azure
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2e451ec4ac41e42f8edf06e1ce87009f9c296c2aace875df12e0d04576359a85
|
|
| MD5 |
bb05f777fcc6b5cbd1ab960dd5ed935e
|
|
| BLAKE2b-256 |
98e24431d52b875689517eb8507fcdcb66cb054b95fd0dcf7b6c0740bf71fdc4
|
