Skip to main content

Amphibious new data transformer to prepare various sources for CGP DSS Data Loader

Project description

newt-transformer

Build Status

Metadata transformer to convert from gen3 to something readable by cgp-dss-data-loader

Common Setup

  1. (optional) We recommend using a Python 3 virtual environment.

  2. Run:

    pip3 install newt-transformer

Setup for Development

  1. Clone the repo:

    git clone https://github.com/jessebrennan/newt-transformer.git

  2. Go to the root directory of the cloned project:

    cd newt-transformer

  3. Run (ideally in a new virtual environment):

    make develop

Running Tests

  1. make sure you followed Setup for Development

  2. Run:

    make test

Transforming data from sheepdog-exporter

  1. The first step is to extract the Gen3 data you want using the sheepdog exporter. The TopMed public data extracted from sheepdog is available on the release page under Assets. Assuming you use this data, you will now have a file called topmed-public.json

  2. Make sure you are running the virtual environment you set up in the Setup instructions.

  3. Now we need to transform the data. From the root of the project run:

    newt new /path/to/topmed-public.json --output-json transformed-topmed-public.json

    This will generate a transformed output file called transformed-topmed-public.json.

    The new argument specifies that we want the most recent version of the transformer output format. It can be replaced with a gen3 argument, but this older format will soon be deprecated.

  4. Likely you want to upload this data to the DSS. Instructions for this can be found at the DSS data loader repo.

Transformer Architecture

see github for details

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
newt-transformer-0.1.7.tar.gz (11.8 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page