acdc-aws-etl-pipeline

Infrastructure and code for the ACDC ETL pipeline and data operations in AWS

Ingestion

DBT

Release Management

Writing DBT Releases

Deploying the dictionary

e.g. to testing

# Example 
bash services/dictionary/pull_dict.sh <raw_dictionary_url>
bash services/dictionary/upload_dictionary.py <local_dictionary_path> <s3_target_uri>

# Deploying to test
VERSION=v0.6.3
bash services/dictionary/pull_dict.sh "https://raw.githubusercontent.com/AustralianBioCommons/acdc-schema-json/refs/tags/${VERSION}/dictionary/prod_dict/acdc_schema.json"
python3 services/dictionary/upload_dictionary.py "services/dictionary/schemas/acdc_schema_${VERSION}.json" s3://gen3schema-cad-uat-biocommons.org.au/cad.json


# Deploying to staging
VERSION=v1.0.0
bash services/dictionary/pull_dict.sh "https://raw.githubusercontent.com/AustralianBioCommons/acdc-schema-json/refs/tags/${VERSION}/dictionary/prod_dict/acdc_schema.json"
python3 services/dictionary/upload_dictionary.py "services/dictionary/schemas/acdc_schema_${VERSION}.json" s3://gen3schema-cad-staging-biocommons.org.au/cad.json

Generating synthetic metadata

Run this script to generate synthetic metadata for the studies in the dictionary

# this will generate 30 samples for AusDiab_Simulated and 60 samples for Baker-Biobank_Simulated
bash services/synthetic_data/generate_synth_metadata.sh --studies "AusDiab_Simulated,Baker-Biobank_Simulated" --permute-max-samples "30,60"

uploading synthetic metadata to sheepdog

Run this script to upload synthetic metadata to sheepdog

# to see argumments
python3 services/synthetic_data/upload_synth_metadata_sheepdog.py -h

# to upload metadata for version v0.6.3
python3 services/synthetic_data/upload_synth_metadata_sheepdog.py --version v0.6.3

Metadata

Release files for acdc_aws_etl_pipeline 0.4.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for acdc_aws_etl_pipeline 0.4.6
File	Size	Uploaded
acdc_aws_etl_pipeline-0.4.6.tar.gz	32.1 kB	Nov 20, 2025	Details

Built distribution (wheel)

Table of built distributions (wheels) for acdc_aws_etl_pipeline 0.4.6
File	Interpreter	ABI	Platform	Reset
acdc_aws_etl_pipeline-0.4.6-py3-none-any.whl 37.3 kB Nov 20, 2025	Python 3	none	any	Details

Total release size: 69.4 kB

Release files / acdc_aws_etl_pipeline-0.4.6.tar.gz

Download URL	acdc_aws_etl_pipeline-0.4.6.tar.gz
Size	32.1 kB
Tags	Source
SHA-256 checksum How to use checksums	`96067eb59b79d8c07234163a5da10aede5dba4af5c19716b9a49ab648316b806`
BLAKE2b-256 checksum How to use checksums	`04d27557689b9fd3fcc412c9d1e4902766e56d804fe4ef35ee18985a78625b94`
Upload date	Nov 20, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/2.2.1 CPython/3.9.25 Linux/6.11.0-1018-azure`

Release files / acdc_aws_etl_pipeline-0.4.6-py3-none-any.whl

Download URL	acdc_aws_etl_pipeline-0.4.6-py3-none-any.whl
Size	37.3 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`3692b6a75cd40c935a7e8b54a95e31ac92ef8421d8757e377491de19874d531f`
BLAKE2b-256 checksum How to use checksums	`075a706bb78142eef1df3e725b18c321aeb07d8151d5ea122bb14184693e8111`
Upload date	Nov 20, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/2.2.1 CPython/3.9.25 Linux/6.11.0-1018-azure`