Skip to main content

VICC normalization routine for genes

Project description

Gene Normalizer

PyPI - Python Version tests DOI

Services and guidelines for normalizing gene terms

Basic usage

Create a database instance and pass it to the QueryHandler constructor:

>>> from gene.query import QueryHandler
>>> from gene.database import create_db
>>> q = QueryHandler(create_db())

Call the normalize() method with a gene term. If available, a rich description of the normalized concept is returned.

>>> result = q.normalize("BRAF")
>>> result.gene_descriptor.gene_id
"hgnc:1097"
>>> result.gene_descriptor.alternate_labels
['NS7', 'RAFB1', 'B-raf', 'BRAF-1', 'BRAF1', 'B-RAF1']

See the documentation for more information, or check out the public REST instance for a live demonstration.

Installation

The Normalizer is available via PyPI:

pip install gene-normalizer[etl,pg]

The [etl,pg] argument tells pip to install packages to fulfill the dependencies of the gene.etl package and the PostgreSQL data storage implementation.

External requirements

Gene Normalization relies on SeqRepo data, which you must download yourself.

From the root directory:

pip install seqrepo
sudo mkdir /usr/local/share/seqrepo
sudo chown $USER /usr/local/share/seqrepo
seqrepo pull -i 2021-01-29  # Replace with latest version using `seqrepo list-remote-instances` if outdated

If you get an error similar to the one below:

PermissionError: [Error 13] Permission denied: '/usr/local/share/seqrepo/2021-01-29._fkuefgd' -> '/usr/local/share/seqrepo/2021-01-29'

You will want to do the following:
(Might not be ._fkuefgd, so replace with your error message path)

sudo mv /usr/local/share/seqrepo/2021-01-29._fkuefgd /usr/local/share/seqrepo/2021-01-29

Use the SEQREPO_ROOT_DIR environment variable to set the path of an already existing SeqRepo directory. The default is /usr/local/share/seqrepo/latest.

Database Initialization

The Normalizer supports two data storage options:

  • DynamoDB, a NoSQL service provided by AWS. This is our preferred storage solution. In addition to cloud deployment, Amazon also provides a tool for local service, which can be installed here. Once downloaded, you can start service by running java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb in a terminal (add a -port <VALUE> option to use a different port)
  • PostgreSQL, a well-known relational database technology. Once starting the Postgres server process, ensure that a database is created (we typically name ours gene_normalizer).

By default, the Gene Normalizer expects to find a DynamoDB instance listening at http://localhost:8000. Alternative locations can be specified in two ways:

The first way is to set the --db_url option to the URL endpoint.

gene_update --update_all --db_url="http://localhost:8001"

The second way is to set the GENE_NORM_DB_URL environment variable to the URL endpoint.

export GENE_NORM_DB_URL="http://localhost:8001"

To use a PostgreSQL instance instead of DynamoDB, provide a PostgreSQL connection URL instead, e.g.

export GENE_NORM_DB_URL="postgresql://postgres@localhost:5432/gene_normalizer"

Adding and refreshing data

Use the gene_update command in a shell to update the database.

Update source(s)

The normalizer currently pulls data from HGNC, Ensembl, and NCBI.

To update one source, simply set --sources to the source you wish to update. The normalizer will check to see if local source data is up-to-date, acquire the most recent data if not, and use it to populate the database.

For example, run the following to acquire the latest HGNC data if necessary, and update the HGNC gene records in the normalizer database:

gene_update --sources="hgnc"

To update multiple sources, you can use the --sources option with the source names separated by spaces.

Update all sources

To update all sources, use the --update_all flag:

gene_update --update_all

Starting the gene normalization service

Once the Gene Normalizer database has been loaded, from the project root, run the following:

uvicorn gene.main:app --reload

Next, view the OpenAPI docs on your local machine:

http://127.0.0.1:8000/gene

Developer instructions

The following sections include instructions specifically for developers.

Installation

For a development install, we recommend using Pipenv. See the pipenv docs for direction on installing pipenv in your compute environment.

Once installed, clone the repo and initialize the environment:

git clone https://github.com/cancervariants/gene-normalization
cd gene-normalization
pipenv shell
pipenv update
pipenv install --dev

Alternatively, install the pg, etl, dev, and test dependency groups in a virtual environment:

git clone https://github.com/cancervariants/gene-normalization
cd gene-normalization
python3 -m virtualenv venv
source venv/bin/activate
pip install -e ".[pg,etl,dev,test]"

Init coding style tests

Code style is managed by flake8 and checked prior to commit.

We use pre-commit to run conformance tests.

This ensures:

  • Check code style
  • Check for added large files
  • Detect AWS Credentials
  • Detect Private Key

Before first commit run:

pre-commit install

Running unit tests

By default, tests will employ an existing database. For test environments where this is unavailable (e.g. in CI), the GENE_TEST environment variable can be set to initialize a local DynamoDB instance with miniature versions of input data files before tests are executed.

export GENE_TEST=true

Running unit tests is as easy as pytest.

pipenv run pytest

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

gene-normalizer-0.1.37.tar.gz (53.1 kB view details)

Uploaded Source

Built Distribution

gene_normalizer-0.1.37-py3-none-any.whl (62.4 kB view details)

Uploaded Python 3

File details

Details for the file gene-normalizer-0.1.37.tar.gz.

File metadata

  • Download URL: gene-normalizer-0.1.37.tar.gz
  • Upload date:
  • Size: 53.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.6

File hashes

Hashes for gene-normalizer-0.1.37.tar.gz
Algorithm Hash digest
SHA256 375f9355ac1250893785114bed2be6d7374b38602bea78354a170ff487c7bbc8
MD5 7833b43265480ebb96a01f3a154a9481
BLAKE2b-256 0fb3c0eb5bccd18d9ff993b327d29d9ffdf7be07b141395a5e0ef4559f9e5ab1

See more details on using hashes here.

File details

Details for the file gene_normalizer-0.1.37-py3-none-any.whl.

File metadata

File hashes

Hashes for gene_normalizer-0.1.37-py3-none-any.whl
Algorithm Hash digest
SHA256 4bae5e2776d88460c0863c9cb4c48cc82fedacd90c1bedfa95c9a5360fa539aa
MD5 d82a202359b6dbbd0002faa5acb8239b
BLAKE2b-256 de9f5eefdf75fd011511f857707b3d27c1dbc7198c04b608eae285b010409107

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page