acdc-aws-etl-pipeline

Infrastructure and code for the ACDC ETL pipeline and data operations in AWS

Ingestion

DBT

Release Management

Deploying the dictionary

e.g. to testing

# Example 
bash services/dictionary/pull_dict.sh <raw_dictionary_url>
bash services/dictionary/upload_dictionary.py <local_dictionary_path> <s3_target_uri>

# Deploying to test
VERSION=v0.6.3
bash services/dictionary/pull_dict.sh "https://raw.githubusercontent.com/AustralianBioCommons/acdc-schema-json/refs/tags/${VERSION}/dictionary/prod_dict/acdc_schema.json"
python3 services/dictionary/upload_dictionary.py "services/dictionary/schemas/acdc_schema_${VERSION}.json" s3://gen3schema-cad-uat-biocommons.org.au/cad.json


# Deploying to staging
VERSION=v1.0.0
bash services/dictionary/pull_dict.sh "https://raw.githubusercontent.com/AustralianBioCommons/acdc-schema-json/refs/tags/${VERSION}/dictionary/prod_dict/acdc_schema.json"
python3 services/dictionary/upload_dictionary.py "services/dictionary/schemas/acdc_schema_${VERSION}.json" s3://gen3schema-cad-staging-biocommons.org.au/cad.json

Generating synthetic metadata

  • Run this script to generate synthetic metadata for the studies in the dictionary
# this will generate 30 samples for AusDiab_Simulated and 60 samples for Baker-Biobank_Simulated
bash services/synthetic_data/generate_synth_metadata.sh --studies "AusDiab_Simulated,Baker-Biobank_Simulated" --permute-max-samples "30,60"

uploading synthetic metadata to sheepdog

  • Run this script to upload synthetic metadata to sheepdog
# to see argumments
python3 services/synthetic_data/upload_synth_metadata_sheepdog.py -h

# to upload metadata for version v0.6.3
python3 services/synthetic_data/upload_synth_metadata_sheepdog.py --version v0.6.3

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.5.4)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for harjo391 from gravatar.com harjo391

Credits

Author: JoshuaHarris391

Requires

Python <4.0.0, >=3.9.5

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

acdc_aws_etl_pipeline-0.6.5.tar.gz (32.4 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

acdc_aws_etl_pipeline-0.6.5-py3-none-any.whl (37.5 kB view details)

Uploaded Python 3

File details

Details for the file acdc_aws_etl_pipeline-0.6.5.tar.gz.

File metadata

  • Download URL: acdc_aws_etl_pipeline-0.6.5.tar.gz
  • Upload date:
  • Size: 32.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/2.2.1 CPython/3.9.25 Linux/6.11.0-1018-azure

File hashes

Hashes for acdc_aws_etl_pipeline-0.6.5.tar.gz
Algorithm Hash digest
SHA256 5cbe2e4fef7c445dbadef59d3627b2faeb613c0a033c264a350766864b776d1b
MD5 0cccfa8bd0949945cbc75b6e7da3137e
BLAKE2b-256 02d2bb3deb31dfc83515c6decf58c644abae654406186454c4399320824e18ad

See more details on using hashes here.

File details

Details for the file acdc_aws_etl_pipeline-0.6.5-py3-none-any.whl.

File metadata

File hashes

Hashes for acdc_aws_etl_pipeline-0.6.5-py3-none-any.whl
Algorithm Hash digest
SHA256 147b9342cf69932bac7c48f6bdd49c8a0943bdd3d56c0f9c4722e7f36ca5634e
MD5 9082bef53d1989aafff9dce8d5277cab
BLAKE2b-256 6c61df14b854dbf9b9561b02f474473f148dad41cf8a6035ded317d0d3aa05e6

See more details on using hashes here.

Release history Release notifications | RSS feed

1.5.4

2 files

1.5.3

2 files

1.5.2

2 files

1.5.1

2 files

1.5.0

2 files

1.2.0

2 files

1.1.0

2 files

1.0.1

2 files

1.0.0

2 files

0.9.9

2 files

0.9.8

2 files

0.9.7

2 files

0.9.6

2 files

0.9.5

2 files

0.9.4

2 files

0.9.3

2 files

0.9.2

2 files

0.9.1

2 files

0.9.0

2 files

0.8.2

2 files

0.8.1

2 files

0.8.0

2 files

0.7.6

2 files

0.7.5

2 files

0.7.4

2 files

0.7.3

2 files

0.7.2

2 files

0.7.1

2 files

0.7.0

2 files

0.6.9

2 files

0.6.8

2 files

0.6.7

2 files

0.6.6

2 files

This release

0.6.5 This release

2 files

0.6.4

2 files

0.6.3

2 files

0.6.2

2 files

0.6.1

2 files

0.6.0

2 files

0.5.9

2 files

0.5.8

2 files

0.5.7

2 files

0.5.6

2 files

0.5.5

2 files

0.5.4

2 files

0.5.3

2 files

0.5.2

2 files

0.5.1

2 files

0.5.0

2 files

0.4.9

2 files

0.4.8

2 files

0.4.7

2 files

0.4.6

2 files

0.4.5

2 files

0.4.3

2 files

0.4.2

2 files

0.4.1

2 files

0.4.0

2 files

0.3.9

2 files

0.3.8

2 files

0.3.7

2 files

0.3.6

2 files

0.3.5

2 files

0.3.4

2 files

0.3.3

2 files

0.3.2

2 files

0.3.1

2 files

0.3.0

2 files

0.2.9

2 files

0.2.8

2 files

0.2.7

2 files

0.2.6

2 files

0.2.5

2 files

0.2.4

2 files

0.2.3

2 files

0.2.2

2 files

0.2.1

2 files

0.2.0

2 files

0.1.9

2 files

0.1.8

2 files

0.1.7

2 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page