acdc-aws-etl-pipeline
Infrastructure and code for the ACDC ETL pipeline and data operations in AWS
Ingestion
DBT
Release Management
Deploying the dictionary
e.g. to testing
# Example
bash services/dictionary/pull_dict.sh <raw_dictionary_url>
bash services/dictionary/upload_dictionary.py <local_dictionary_path> <s3_target_uri>
# Deploying to test
VERSION=v0.6.3
bash services/dictionary/pull_dict.sh "https://raw.githubusercontent.com/AustralianBioCommons/acdc-schema-json/refs/tags/${VERSION}/dictionary/prod_dict/acdc_schema.json"
python3 services/dictionary/upload_dictionary.py "services/dictionary/schemas/acdc_schema_${VERSION}.json" s3://gen3schema-cad-uat-biocommons.org.au/cad.json
# Deploying to staging
VERSION=v1.0.0
bash services/dictionary/pull_dict.sh "https://raw.githubusercontent.com/AustralianBioCommons/acdc-schema-json/refs/tags/${VERSION}/dictionary/prod_dict/acdc_schema.json"
python3 services/dictionary/upload_dictionary.py "services/dictionary/schemas/acdc_schema_${VERSION}.json" s3://gen3schema-cad-staging-biocommons.org.au/cad.json
Generating synthetic metadata
- Run this script to generate synthetic metadata for the studies in the dictionary
# this will generate 30 samples for AusDiab_Simulated and 60 samples for Baker-Biobank_Simulated
bash services/synthetic_data/generate_synth_metadata.sh --studies "AusDiab_Simulated,Baker-Biobank_Simulated" --permute-max-samples "30,60"
uploading synthetic metadata to sheepdog
- Run this script to upload synthetic metadata to sheepdog
# to see argumments
python3 services/synthetic_data/upload_synth_metadata_sheepdog.py -h
# to upload metadata for version v0.6.3
python3 services/synthetic_data/upload_synth_metadata_sheepdog.py --version v0.6.3
2 maintainers
Metadata
2 maintainers
Release files for acdc_aws_etl_pipeline 0.4.9
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| acdc_aws_etl_pipeline-0.4.9.tar.gz | 32.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| acdc_aws_etl_pipeline-0.4.9-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 69.6 kB
Release files / acdc_aws_etl_pipeline-0.4.9.tar.gz
| Download URL | acdc_aws_etl_pipeline-0.4.9.tar.gz |
|---|---|
| Size | 32.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6cd41a6833a65cf1658978cffa287e1b90ffbdfec7aa24a7bcbf4628d2e66f7d
|
|
BLAKE2b-256 checksum How to use checksums |
52a08c5cdc5750222975af3ba99c0865d1d801afd034b07d45de79e433a681f9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.2.1 CPython/3.9.25 Linux/6.11.0-1018-azure
|
Release files / acdc_aws_etl_pipeline-0.4.9-py3-none-any.whl
| Download URL | acdc_aws_etl_pipeline-0.4.9-py3-none-any.whl |
|---|---|
| Size | 37.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ce8ff9c1313076d5e46b076295567cfa3e6f8642da184222c92c873420ef4bb5
|
|
BLAKE2b-256 checksum How to use checksums |
d738449a680a1d0317e59d55823f5e9b3e5ea6474d9ed5636c6fcc3ff75acd5d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.2.1 CPython/3.9.25 Linux/6.11.0-1018-azure
|
