Utilities for managing nlp models and for processing text-related data at Wellcome Data Labs
Project description
WellcomeML utils
This package contains common utility functions for usual tasks at Wellcome Data Labs, in particular functionalities for processing, embedding and classifying text data. This includes
- An intuitive sklearn-like API wrapping text vectorizers, such as Doc2vec, Bert, Scibert
- Common API for off-the-shelf classifiers to allow quick iteration (e.g. Frequency Vectorizer, Bert, Scibert, basic CNN, BiLSTM)
- Utils to download and convert academic text datasets for benchmark
For more information read the official
1. Quickstart
Installing from PyPi
pip install wellcomeml
This will install the "vanilla" package. In order to install the deep-learning functionality (torch/transformers/spacy transformers):
pip install wellcomeml[deep-learning]
Installing from a release wheel
Download the wheel from aws and pip install it:
pip install wellcomeml-2020.1.0-py3-none-any.whl
pip install wellcomeml-2020.1.0-py3-none-any.whl[deep-learning]
1.1 Installing wellcomeml[deep-learning] on windows
Torch has a different installation for windows so it will not get automatically installed with wellcomeml[deeplearning]. It needs to be installed first (this is for machines with no CUDA parallel computing platform for those that do look here https://pytorch.org/ for correct installation):
pip install torch==1.5.1+cpu torchvision==0.6.1+cpu -f https://download.pytorch.org/whl/torch_stable.html
Then install wellcomeml[deep-learning]:
pip install wellcomeml[deep-learning]
2. Development
2.1 Build local virtualenv
make
2.2 Contributing to the docs
Make changes to the .rst files in /docs
(please do not change the ones starting by wellcomeml as those are generated automatically)
Navigate to the root repository and run
make update-docs
Verify that _build/html/index.html
has generated correctly and submit a PR.
2.3 Release a new version (and upload to aws s3/pypi/github)
First create a github token, if you haven't one, with artifact write access and export it to the env variables:
export GITHUB_TOKEN=...
The checklist for a new release is:
- Change
wellcomeml/__version__.py
- Add changelog
-
make dist
- Verify new package was generated correctly on the pip registry and GitHub releases
2.4 (Optional) Installing from other locations
pip3 install <relative path to this folder>
2.5 Transformers
On OSX, if you get a message complaining about the rust compiler, install and initialise it with:
brew install rustup
rustup-init
3. Example usage of some modules
Examples can be found in the subfolder examples
.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for wellcomeml-1.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0e2ccd2bcff9fb43825ef1927c9114d9e46f919c75f3623358702afad026c5ae |
|
MD5 | 8d3065bcba1e01e872ce759a42de3fec |
|
BLAKE2b-256 | 415fe260cb9d62082ef9f2df561b0eb5373a630c12eefa1e3547cd36a3705535 |