Skip to main content

Amphibious new data transformer to prepare various sources for CGP DSS Data Loader

Project description


Build Status

Metadata transformer to convert from gen3 to something readable by cgp-dss-data-loader

Common Setup

  1. (optional) We recommend using a Python 3 virtual environment.

  2. Run:

    pip3 install newt-transformer

Setup for Development

  1. Clone the repo:

    git clone

  2. Go to the root directory of the cloned project:

    cd newt-transformer

  3. Run (ideally in a new virtual environment):

    make develop

Running Tests

  1. make sure you followed Setup for Development

  2. Run:

    make test

Transforming data from sheepdog-exporter

  1. The first step is to extract the Gen3 data you want using the sheepdog exporter. The TopMed public data extracted from sheepdog is available on the release page under Assets. Assuming you use this data, you will now have a file called topmed-public.json

  2. Make sure you are running the virtual environment you set up in the Setup instructions.

  3. Now we need to transform the data. From the root of the project run:

    newt new /path/to/topmed-public.json --output-json transformed-topmed-public.json

    This will generate a transformed output file called transformed-topmed-public.json.

    The new argument specifies that we want the most recent version of the transformer output format. It can be replaced with a gen3 argument, but this older format will soon be deprecated.

  4. Likely you want to upload this data to the DSS. Instructions for this can be found at the DSS data loader repo.

Transformer Architecture

see github for details

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for newt-transformer, version 0.1.7
Filename, size File type Python version Upload date Hashes
Filename, size newt-transformer-0.1.7.tar.gz (11.8 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page