AnVIL client library. Combines gen3, terra client APIs with single signon and data harmonization use cases.
Project description
pyAnVIL: terra + gen3
A python client integration of gen3 and terra.
For python developers, who have requirements to access both terra and gen3 platforms, pyAnVIL is an integration module that provides SSO (single sign on) using terra as an IDP (identity provider) and manages distribution of dependencies unlike juggling multiple credentials and installs, pyAnVIL provides developer friendly experience.
Installation
Pre-requisites:
-
gcloud cli tools installed and configured gcloud.
-
Google Id provisioned in both Terra and Gen3:
- One time Account Linking:
- Pre-requisite: google account provisioned in both Gen3 and Terra.
- Log into https://gen3.theanvil.io/
- Log into https://anvil.terra.bio
- In Terra, navigate to your profile
- Under "IDENTITY & EXTERNAL SERVERS", log into
NHGRI AnVIL Data Commons Framework Services
, the system should present you with a Gen3 Oauth flow. - Note the google project used for billing
- Under "IDENTITY & EXTERNAL SERVERS", log into
- One time Account Linking:
-
Per instance, terra API setup:
- Use the google account and billing project to setup credentials for the terra api.
gcloud auth login <google-account> gcloud auth application-default set-quota-project <billing-project-id>
- Use the google account and billing project to setup credentials for the terra api.
-
Validation
gcloud auth print-access-token >>> ya29.a0AfH6SMBSPFSt252qQNl....... fissfc config >>> .... root_url https://broad-bond-prod.appspot.com/
-
Setup
pip install pyAnVIL
Use cases
SSO
from anvil.gen3_auth import Gen3TerraAuth
from gen3.submission import Gen3Submission
auth = Gen3TerraAuth()
gen3_endpoint = "https://gen3.theanvil.io"
submission_client = Gen3Submission(gen3_endpoint, auth)
API Wrappers
Gen3
query = '{project(first:0) {code, subjects {submitter_id}, programs {name} }}'
results = submission_client.query(query)
[p['code'] for p in results['data']['project']]
>>> ['GTEx', '1000Genomes']
Terra
from anvil.terra import FAPI
FAPI.whoami()
>>> 'anvil.user@gmail.com'
Terra utilities
from anvil.terra.api import get_projects
projects = get_projects(namespaces=['anvil-datastorage'], project_pattern='AnVIL_CCDG.*')
[p['workspace']['name'] for p in projects]
>>> ['AnVIL_CCDG_WashU_CVD_EOCAD_BioMe_WGS',
'AnVIL_CCDG_Broad_CVD_EOCAD_TaiChi_WGS',
'AnVIL_CCDG_Broad_AI_IBD_Brant_DS-IBD_WGS', ...]
Data Dashboard
Configuration
The following environment variable can be set in order to configure library functionality
environment | example | default | description |
---|---|---|---|
PYANVIL_CACHE_PATH | db.sqlite3 | /tmp/pyanvil-cache.sqlite | Configure where you want the pyAnVIL cache to be |
Contributing
-
set up virtual env
python3 -m venv venv source venv/bin/activate python3 -m pip install -r requirements.txt python3 -m pip install -r requirements-dev.txt
-
test gen3 authorization
python3 -m pytest --user_email <GMAIL ACCOUNT> --log-level DEBUG --gen3_endpoint <GEN3_ENDPOINT> tests/integration/test_gen3_auth.py
-
continuous integration
# see https://github.com/broadinstitute/firecloud-tools/tree/master/scripts/register_service_account docker run --rm -it -v "$HOME"/.config:/.config -v /Users/walsbr/client-apis/pyAnVIL/client_secret.json:/svc.json broadinstitute/firecloud-tools python /scripts/register_service_account/register_service_account.py -j /svc.json -e brian@bwalsh.com The service account pyanvil@api-project-807881269549.bwalsh.com.iam.gserviceaccount.com is now registered with FireCloud. You can share workspaces with this address, or use it to call APIs.
Distribution
- PyPi
# refresh from data ingestion tracker spread sheet, update pypi
export TWINE_USERNAME= # the username to use for authentication to the repository.
export TWINE_PASSWORD= # the password to use for authentication to the repository.
rm -r dist/
python3 setup.py data_ingestion_tracker sdist bdist_wheel
twine upload dist/*
- Read The Docs
https://readthedocs.org/projects/pyanvil/
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pyAnVIL-0.0.6rc3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ba096a6644f6ee0fddec66298a9cd729086795de6e769a0da5c13a7e9579a1d6 |
|
MD5 | 9b0366f86eab6a96a269cd03a8b86c15 |
|
BLAKE2b-256 | afdfaaf08609e72d0c8d37a6f874f3b9a615c76896e73351ec2ad874124010d3 |