thoipapy 2.0.0

How does thoipapy work?

• downloads protein homologues with BLAST

• extracts residue properties (e.g. residue conservation and polarity)

• trains a machine learning classifier

• validates the prediction performance

• creates heatmaps of residue properties and THOIPA prediction

Installation

THOIPA is on PyPI, but read this before installing it.

pip install "thoipapy>=2.1"

Ask for 2.1 or newer. Releases up to 1.2.0 were published between 2018 and 2021 and no longer work: the model they ship is a scikit-learn 0.23 pickle, which no current scikit-learn can load, so they install without error and then fail on the first prediction.

Install it into a new, empty environment. Every dependency is pinned to an exact version (numpy==2.5.2, pandas==3.0.5, and so on), so pip will almost certainly refuse to install it alongside packages you already have. That is intended, not a defect: THOIPA is not actively maintained, and exact pins are what stop an unattended install from silently resolving to a different numerical result years from now.

The same pinning means that on a future python, where those exact versions have no wheel, the install will fail outright rather than succeed and misbehave. If that happens, use the conda environment below, which also pins the interpreter.

conda env create -f environment.yml
conda activate thoipapy
pip install -e .

This is the supported path, and the only one that pins python itself along with the external command-line tools.

THOIPA has only been tested on Linux, because of its reliance on external programs such as FreeContact, CD-HIT and rate4site.

Dependencies

THOIPApy requires python 3.12 or newer and is tested on python 3.13.

Every dependency is pinned to an exact version. This is deliberate. THOIPA is scientific software whose output has to stay reproducible, and a version range resolves to something different every time it is installed. The failure that causes is not a helpful error at install time; it is a prediction that quietly differs from the published one. Pinning trades the ability to install alongside arbitrary other packages for the ability to still work in five years. Use a dedicated environment.

Pins live in three places, which are kept in step:

The external tools are pinned for the same reason as the python packages: a different rate4site or cd-hit changes the conservation scores and the redundancy reduction, and therefore the features and the prediction.

You are of course free to relax the pins and test a newer combination. Nothing guarantees that it resolves, or that predictions still match. If you do, run the full test suite, including the golden-file regression tests, before trusting the result.

Several pipeline steps call external command-line programs rather than python libraries. blast, cd-hit and rate4site are installed by environment.yml. freecontact is not packaged for conda on any channel, so it is installed separately, either with root:

sudo apt-get install freecontact

or without, into the active conda environment:

./scripts/install_freecontact.sh

Phobius is no longer required. It was needed only for the n_TMDs feature, which was removed in 2.0.0. See docs/n_TMDs_dropped.md.

Getting the data (training sets, models, test data)

If you only want to run predictions, you do not need any of this. The trained model ships inside the package, so pip install thoipapy gives you a working predictor with no data download, no clone, and no DVC. Everything below is for developers who want to retrain the model, re-run the validation, or run the test suite.

The code lives in git; the data does not. data/ is about 128 MB of training sets, homologue alignments, extracted features and prediction outputs, which is too large and too binary to keep in a git repository. It is stored separately and fetched with a tool called DVC (Data Version Control).

If you have not used DVC before, the idea is simple. DVC leaves a small text file in git for each data directory, for example data/features.dvc. That file records a checksum, not the data. When you run dvc pull, DVC reads those checksums and downloads the matching files from a data store into data/. It is, in effect, git clone for large files. You do not need an account, a login, or any credentials to fetch this project’s data.

Full documentation: https://dvc.org/doc . A good short introduction is https://dvc.org/doc/start/data-management/data-versioning .

# with conda or mamba (recommended, matches environment.yml)
conda install -c conda-forge dvc dvc-http

# or with pip
pip install "dvc[http]"

dvc --version

dvc pull -j 4

dvc status

dvc pull -j 4 test/regression_data.dvc  # run the test suite                    28 KB
dvc pull -j 4 data/features.dvc         # retrain the model                      18 MB
dvc pull -j 4 data/results.dvc          # trained models and validation output   25 MB
dvc pull -j 4 data/homologues.dvc       # re-extract features from alignments    60 MB

Running BLAST locally instead of at NCBI

Recommended: run THOIPA against a local UniRef90 database rather than querying NCBI.

mamba install -c bioconda blast

# Downloads UniRef90 and runs makeblastdb. Idempotent: an existing download is not refetched.
# Expect several hours for the download and 1-2 hours to build.
python scripts/build_local_blast_db.py

export THOIPA_LOCAL_BLAST_DB=/mnt/shared/blastdb/uniref90

wget https://ftp.uniprot.org/pub/databases/uniprot/uniref/uniref90/uniref90.fasta.gz
gunzip uniref90.fasta.gz
makeblastdb -in uniref90.fasta -dbtype prot -out uniref90 -title uniref90 -parse_seqids

Reproducing the published results

Versions from 2.0.0 onwards include corrections and improvements made after publication, so their output differs slightly from the numbers in the 2020 paper. The differences are small and do not change the conclusions, but they are real.

To reproduce the results as published, use the state of the repository at the time of publication rather than the current release:

• thoipapy v1.2.0, git tag 1.2.0 (commit 67a1438)

• the dataset published via the Open Science Foundation

Note that v1.2.0 requires python 3.8 and the dependency versions pinned in its requirements.txt; it will not run on a current scientific python stack.

For any new work, use the current release. The main post-publication corrections are that feature selection is now deterministic (it previously depended on the interpreter’s hash seed, so the selected feature set varied between runs), model training and hyperparameter tuning are seeded, and several ineffective filters and dead assertions were repaired.

Usage as a standalone predictor

• for local predictions on linux, first install NCBI_BLAST, biopython, freecontact, CD-HIT and rate4site

• see test/functional/test_standalone_prediction.py for the current run syntax, typically

from thoipapy import run_THOIPA_prediction
from thoipapy.predict import get_md5_checksum
from thoipapy.utils import make_sure_path_exists

protein_name = "ERBB3"
TMD_seq = "MALTVIAGLVVIFMMLGGTFL"
full_seq = "MVQNECRPCHENCTQGCKGPELQDCLGQTLVLIGKTHLTMALTVIAGLVVIFMMLGGTFLYWRGRRIQNKRAMRRYLERGESIEPLDPSEKANKVLA"
out_dir = "/path/to/your/desired/output/folder"
make_sure_path_exists(out_dir)
md5 = get_md5_checksum(TMD_seq, full_seq)
run_THOIPA_prediction(protein_name, md5, TMD_seq, full_seq, out_dir)

Example Output

• the output includes a csv showing the THOIPA prediction for each residue, as well as a heatmap figure as a summary

• below is a heatmap showing the THOIPA prediction, and underlying conservation, relative polarity, and coevolution

Create your own machine learning predictor

• THOIPA can be retrained to any dataset of your choice

• the original set of training sequences and other resources are published via the Open Science Foundation, and are also what dvc pull fetches into data/

• the THOIPA feature extraction, feature selection, and training pipeline is fully automated

• open an issue for an introduction to the THOIPA software pipeline and settings

The data, sets and base directories are resolved from the repository itself (see thoipapy/paths.py); they are no longer read from the settings spreadsheet. Protein sets are defined in data/sets/. Because the training data is not shipped inside the installed package, the training pipeline can only be run from a checkout of this repository, not from pip install thoipapy.

Where things are

Settings are CSV. thoipapy/setting/run_settings_example.csv drives a training run, standalone_run_settings.csv a single prediction, and model_features.csv lists the features eligible for the model. Each row of the two run-settings files carries a type column and is converted on load.

Pipeline functions take explicit named parameters rather than a settings dictionary, so a signature states what the function reads:

def lipo_from_pssm_mult_prot(paths, df_set, lipophilicity_scale, logging):

Artefact filenames encode the settings that produced them (surr20.gaps5). ArtefactPaths computes them, so writers and readers cannot disagree when a setting changes.

Run the training pipeline with:

python -m thoipapy.run

The settings path is hardcoded in run.py. To use a different one, import run() and pass your own settings dict.

License

THOIPApy is free software distributed under the permissive MIT License.

Contribute

• Contributors are welcome.

• For bug reports, questions and feature requests, please open an issue.

Contact

THOIPA is maintained by Mark Teese. To make contact, open an issue or use LinkedIn.

• Mark Teese, 22DataCatalysis GmbH, formerly of the Langosch Lab at the Technical University of Munich

• Bo Zeng, Chinese Academy of Sciences, Beijing, formerly of the Frishman Lab at the Technical University of Munich

Citation

Yao Xiao, Bo Zeng, Nicola Berner, Dmitrij Frishman, Dieter Langosch, and Mark George Teese (2020) Experimental determination and data-driven prediction of homotypic transmembrane domain interfaces, Computational and Structural Biotechnology Journal