ProteinGymPy

Overview

ProteinGym comprises a collection of benchmarks for evaluating the performance of models predicting the effect of point mutations generated by Notin et al., 2023. These datasets include Deep Mutational Scanning Assays (DMS) for 186 proteins, as well as performance scores for several models in both zero-shot and semi-supervised settings.

ProteinGymPy provides analysis-ready data resources from ProteinGym (Notin et al., 2023) and built-in functionality to visualize the data in Python. ProteinGym comprises a collection of benchmarks for evaluating the performance of models predicting the effect of point mutations. This package provides access to:

1. Deep mutational scanning (DMS) scores from 217 assays measuring the impact of all possible amino acid substitutions across 186 proteins, and
2. Model performance metrics and prediction scores from 79 variant prediction models in a zero-shot setting and 12 models in a semi-supervised setting.

Installation

To install the software, we recommend using uv for python package management:

uv venv --python=3.13  #create a venv
source .venv/bin/activate 
uv pip install `ProteinGymPy`

Addtional packages are required for jupyter notebook visualization functions:

uv pip install 'ProteinGymPy[visualization]`

To pull in the latest commits directly from github, use the following:

uv pip install 'ProteinGymPy[visualization] @ git+https://github.com/ccb-hms/ProteinGymPy.git'

Usage

Quick Start

Load DMS substitution data (217 assays):

from proteingympy import get_dms_substitution_data

# Load all DMS assays with UniProt IDs  
dms_data = get_dms_substitution_data()
print(f"Loaded {len(dms_data)} DMS assays")

# Access specific assay
assay_name = list(dms_data.keys())[0]
df = dms_data[assay_name]
print(df.head())

Load other datasets:

from proteingympy import (
    get_alphamissense_proteingym_data,
    get_zero_shot_metrics,
    get_supervised_substitution_data
)

# AlphaMissense pathogenicity scores
am_data = get_alphamissense_proteingym_data()

# Zero-shot benchmarking metrics  
benchmarks = get_zero_shot_metrics()
print(f"Available metrics: {list(benchmarks.keys())}")

# Supervised model predictions
supervised_data, summary = get_supervised_substitution_data("random_5")

Available Functions

Running Examples

Run the full example script (includes data downloads):

source .venv/bin/activate
python examples/proteingym_pipeline_examples.py

This demonstrates all available data loading functions and shows the structure of each dataset.

Running Tests

Run the test suite to verify functionality:

source .venv/bin/activate
python -m pytest tests/ -v

Or run specific test files:

source .venv/bin/activate  
python tests/test_data_pipelines.py
python tests/test_basic.py

The tests include:

• Unit tests for all data loading functions
• Integration tests for complete workflows
• Mock tests that don't require network access
• Validation of data structure and consistency

Documentation building

This project uses Mkdocs as a project document builder. Static docs can be built by installing the extra dependencies after cloning the repo

#git clone https://github.com/ccb-hms/ProteinGymPy.git
cd ProteinGymPy
uv sync --extra mkdocs
uv run mkdocs build

Citation

Notin, P., A. Kollasch, D. Ritter, L. van Niekerk, S. Paul, H. Spinner, N. Rollins, et al. 2023. “ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design.” In Advances in Neural Information Processing Systems, edited by A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, 36:64331–79. Curran Associates, Inc.