Hestia-GOOD

Computational tool for generating generalisation-evaluating evaluation sets.

• Documentation: https://ibm.github.io/Hestia-GOOD
• Source Code: https://github.com/IBM/Hestia-GOOD
• Paper [ICLR 2025]: https://openreview.net/pdf?id=qFZnAC4GHR

Contents

Installation

Installing in a conda environment is recommended. For creating the environment, please run:

conda create -n hestia python
conda activate hestia

1. Python Package

1.1. From PyPI

pip install hestia-good

1.2. Directly from source

pip install git+https://github.com/IBM/Hestia-GOOD

2. Optional dependencies

2.1. Molecular similarity

RDKit is a dependency necessary for calculating molecular similarities:

pip install rdkit

2.2. Sequence alignment

• MMSeqs2 https://github.com/steineggerlab/mmseqs2

# static build with AVX2 (fastest) (check using: cat /proc/cpuinfo | grep avx2)
wget https://mmseqs.com/latest/mmseqs-linux-avx2.tar.gz; tar xvfz mmseqs-linux-avx2.tar.gz; export PATH=$(pwd)/mmseqs/bin/:$PATH

# static build with SSE4.1  (check using: cat /proc/cpuinfo | grep sse4)
wget https://mmseqs.com/latest/mmseqs-linux-sse41.tar.gz; tar xvfz mmseqs-linux-sse41.tar.gz; export PATH=$(pwd)/mmseqs/bin/:$PATH

# static build with SSE2 (slowest, for very old systems)  (check using: cat /proc/cpuinfo | grep sse2)
wget https://mmseqs.com/latest/mmseqs-linux-sse2.tar.gz; tar xvfz mmseqs-linux-sse2.tar.gz; export PATH=$(pwd)/mmseqs/bin/:$PATH

# MacOS
brew install mmseqs2  

To use Needleman-Wunch, either:

conda install -c bioconda emboss

or

sudo apt install emboss

• Windows: Download binaries from EMBOSS and MMSeqs2-latest

2.3. Structure alignment

• To use Foldseek https://github.com/steineggerlab/foldseek:

# Linux AVX2 build (check using: cat /proc/cpuinfo | grep avx2)
wget https://mmseqs.com/foldseek/foldseek-linux-avx2.tar.gz; tar xvzf foldseek-linux-avx2.tar.gz; export PATH=$(pwd)/foldseek/bin/:$PATH

# Linux SSE2 build (check using: cat /proc/cpuinfo | grep sse2)
wget https://mmseqs.com/foldseek/foldseek-linux-sse2.tar.gz; tar xvzf foldseek-linux-sse2.tar.gz; export PATH=$(pwd)/foldseek/bin/:$PATH

# Linux ARM64 build
wget https://mmseqs.com/foldseek/foldseek-linux-arm64.tar.gz; tar xvzf foldseek-linux-arm64.tar.gz; export PATH=$(pwd)/foldseek/bin/:$PATH

# MacOS
wget https://mmseqs.com/foldseek/foldseek-osx-universal.tar.gz; tar xvzf foldseek-osx-universal.tar.gz; export PATH=$(pwd)/foldseek/bin/:$PATH

Documentation

1. DatasetGenerator

The HestiaGenerator allows for the easy generation of training/validation/evaluation partitions with different similarity thresholds. Enabling the estimation of model generalisation capabilities. It also allows for the calculation of the AU-GOOD (Area Under the Generalization Out-Of-Distribution curve). More information in Dataset Generator docs.

from hestia.dataset_generator import HestiaGenerator, SimArguments

# Initialise the generator for a DataFrame
generator = HestiaGenerator(df)

# Define the similarity arguments (for more info see the documentation page https://ibm.github.io/Hestia-OOD/datasetgenerator)

# Similarity arguments for protein similarity
prot_args = SimArguments(
    data_type='sequence', field_name='sequence',
    alignment_algorithm='mmseqs2+prefilter', verbose=3
)

# Similarity arguments for molecular similarity
mol_args = SimArguments(
    data_type='small molecule', field_name='SMILES',
    fingeprint='mapc', radius=2, bits=2048
)

# Calculate the similarity
generator.calculate_similarity(prot_args)

# Calculate partitions
generator.calculate_partitions(min_threshold=0.3,
                               threshold_step=0.05,
                               test_size=0.2, valid_size=0.1)

# Save partitions
generator.save_precalculated('precalculated_partitions.gz')

# Load pre-calculated partitions
generator.from_precalculated('precalculated_partitions.gz')

# Training code (filter partitions with test sets less than 18.5% of total data)

for threshold, partition in generator.get_partitions(filter=0.185):
    train = df.iloc[partition['train']]
    valid = df.iloc[partition['valid']]
    test = df.iloc[partition['test']]

# ...

# Calculate AU-GOOD
generator.calculate_augood(results, 'test_mcc')

# Plot GOOD
generator.plot_good(results, 'test_mcc')

# Compare two models
results = {'model A': [values_A], 'model B': [values_B]}
generator.compare_models(results, statistical_test='wilcoxon')

2. Similarity calculation

Calculating pairwise similarity between the entities within a DataFrame df_query or between two DataFrames df_query and df_target can be achieved through the calculate_similarity function. More details about similarity calculation can be found in the Similarity calculation documentation.

from hestia.similarity import sequence_similarity_mmseqs
import pandas as pd

df_query = pd.read_csv('example.csv')

# The CSV file needs to have a column describing the entities, i.e., their sequence, their SMILES, or a path to their PDB structure.
# This column corresponds to `field_name` in the function.

sim_df = sequence_similarity_mmseqs(df_query, field_name='sequence', prefilter=True)

3. Clustering

Clustering the entities within a DataFrame df can be achieved through the generate_clusters function. There are three clustering algorithms currently supported: CDHIT, greedy_cover_set, or connected_components. More details about clustering can be found in the Clustering documentation.

from hestia.similarity import sequence_similarity_mmseqs
from hestia.clustering import generate_clusters
import pandas as pd

df = pd.read_csv('example.csv')
sim_df = sequence_similarity_mmseqs(df, field_name='sequence')
clusters_df = generate_clusters(df, field_name='sequence', sim_df=sim_df,
                                cluster_algorithm='CDHIT')

4. Partitioning

Partitioning the entities within a DataFrame df into a training and an evaluation subsets can be achieved through 4 different functions: ccpart, graph_part, reduction_partition, and random_partition. More details about partitioning algorithms can be found in the Partitioning documentation. An example of how ccpart would be used is:

from hestia.similarity import sequence_similarity_mmseqs
from hestia.partition import ccpart
import pandas as pd

df = pd.read_csv('example.csv')
sim_df = sequence_similarity_mmseqs(df, field_name='sequence')
train, test, partition_labs = ccpart(df, threshold=0.3, test_size=0.2, sim_df=sim_df)

train_df = df.iloc[train, :]
test_df = df.iloc[test, :]

5. AutoHestia

AutoHestia automatically benchmarks all combinations of partitioning algorithms and similarity metrics on your dataset, then selects the best-performing splits using guardrails (minimum test size and dynamic range). More information in the AutoHestia docs.

from hestia.autohestia import AutoHestia
import pandas as pd
import numpy as np

df = pd.read_csv('example.csv')

# Pre-computed feature matrix and labels
x = np.load('features.npy')
y = np.load('labels.npy')

# Pre-computed similarity DataFrames keyed by metric name
sim_dfs = {
    'mmseqs2': sim_df_mmseqs,
    'tanimoto': sim_df_tanimoto,
}

auto = AutoHestia(
    df=df,
    field_name='sequence',
    x=x,
    y=y,
    sim_dfs=sim_dfs,
    verbose_level='info'
)

# Run guardrailed experiment sweep
output = auto.best_guardrailed_splits(
    top_k_parts=3,
    top_l_sims=3,
    min_test_size=0.185,
    min_dynamic_range=0.4,
    save_dir='results'
)

# Retrieve the best partitions
best_parts = output['best-parts']
top_combination = output['top-combination']   # (part_alg, sim_metric)

# Plot GOOD curves for all evaluated combinations
auto.plot_good_curves(save_dir='results')

License

Hestia is an open-source software licensed under the MIT Clause License. Check the details in the LICENSE file.