Skip to main content

Simple data loader for CGP HCA Data Store

Project description

cgp-dss-data-loader

Simple data loader for CGP HCA Data Store

Common Setup

  1. (optional) We recommend using a Python 3 virtual environment.

  2. Run:

    pip3 install cgp-dss-data-loader

Setup for Development

  1. Clone the repo:

    git clone https://github.com/DataBiosphere/cgp-dss-data-loader.git

  2. Go to the root directory of the cloned project:

    cd cgp-dss-data-loader

  3. Make sure you are on the branch develop.

  4. Run (ideally in a new virtual environment):

    make develop

Cloud Credentials Setup

Because this program uses Amazon Web Services and Google Cloud Platform, you will need to set up credentials for both of these before you can run the program.

AWS credentials

  1. If you haven't already you will need to make an IAM user and create a new access key. Instructions are here.

  2. Next you will need to store your credentials so that Boto can access them. Instructions are here.

GCP credentials

  1. Follow the steps here to set up your Google Credentials.

Running Tests

Run:

make test

Getting Data from Gen3 and Loading it

  1. The first step is to extract the Gen3 data you want using the sheepdog exporter. The TopMed public data extracted from sheepdog is available on the release page under Assets. Assuming you use this data, you will now have a file called topmed-public.json

  2. Make sure you are running the virtual environment you set up in the Setup instructions.

  3. Now you will need to transform the data into the 'standard' loader format. Do this using the newt-transformer. You can follow the common setup, then the section for transforming data from sheepdog.

  4. Now that we have our new transformed output we can run it with the loader.

    If you used the standard transformer use the command:

    dssload --no-dry-run --dss-endpoint MY_DSS_ENDPOINT --staging-bucket NAME_OF_MY_S3_BUCKET transformed-topmed-public.json
    
  5. You did it!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for cgp-dss-data-loader, version 1.1.0
Filename, size File type Python version Upload date Hashes
Filename, size cgp-dss-data-loader-1.1.0.tar.gz (16.1 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page