Mosses - Model Assessment Toolkit

Description

Mosses is a library that provides a set of functionalities to assess molecular property prediction models, e.g., QSAR/QSPR models. The library currently includes:

• Predictive Validity Module (predictive_validity.py) - Built on top of the concept of predictive validity described by Scannell et al. Nat Rev Drug Discov. 2022;21(12):915-931. doi:10.1038/s41573-022-00552-x. The function predictive_validity.evaluate_pv() allows the analysis of the quality of predictions on a given data set (e.g., a prospective test set of compounds), according to a desired threshold.

  As part of this, evaluate_pv() tracks how similar each month's prospective compounds are to everything predicted before them (model versions released in the same calendar month are pooled), along three axes: similarity of the data, similarity of the correlations, and similarity of the structures. The structural similarity is the mean, over the prospective compounds, of the Tanimoto similarity to their nearest neighbour among the earlier compounds, using Morgan fingerprints (radius 2 over 2048 bits, i.e. ECFP4). It is computed whenever the data carries a structure column -- a column named Structure, Smiles, Canonical SMILES or canonical_smiles (case-insensitive) is recognised automatically, and any other name can be passed as structure_column -- and is simply left out of the plot otherwise. Because it depends only on the compounds and not on the model, it separates "the model is being asked about new chemistry" from "the model is doing worse on chemistry it already knows".

  R² is reported alongside, using the same definition as the R² of the predicted-vs-experimental scatter plot and counting only the prospective compounds: per month in the same similarity plot, and cumulatively -- over all prospective compounds up to each month -- in the model performance section, where the curve ends at the scatter-plot R². R² of a small monthly batch is unstable; the cumulative view shows how many compounds it takes to reach a stable value.

  The similarity chart can also be drawn on its own with Plotter.plot_time_weighted_scores(), e.g. for a figure: optional settings leave out the time-weighted curves and R², add the RMSE of the same compounds (computed by compute_rmse_over_time()) and set how many x labels are shown.

• Heatmap Module (heatmap.py) - Summarises the information from the validation using predictive validity. The heatmap shows in one table, for each series in the data and according to the selected experimental threshold (SET), what the PPV and FOR percentages are, the recommended thresholds and resulting optimised PPV and FOR percentages, as well as, a qualitative label indicating whether the model is Good, Medium, or Bad.

• Multi-Parameter Optimization (MPO) Module (mpo.py) - Provides a comprehensive toolkit for computing and optimizing MPO scores. MPO combines multiple molecular properties into a single score using sigmoid-based desirability functions.

• Data API Module (data_api.py) - Headless counterpart of the evaluations. data_api.predictive_validity_metrics() returns the numbers that evaluate_pv() prints and plots as JSON-serialisable Python primitives, without rendering anything, so they can be served over a REST API or consumed by other programs.

Software Requirements

The library is written in Python and requires a version >= 3.10 for runtime. The dependencies required by the library are defined in pyproject.toml and are automatically installed when installing the library.

How to Install mosses

You can install the library using pip install mosses, directly from GitHub using pip install git+https://github.com/AstraZeneca/mosses.git, or you can clone this repository then run pip install ..

Examples of Usage

Jupyter notebooks with examples can be found in the folder examples. We recommend following those to adapt your data, configs, and code to work with the modules in mosses.

examples/similarity_over_time.ipynb draws the similarity chart of evaluate_pv() on its own, as in the report and with the optional settings for a figure.

Multi-Parameter Optimization (MPO) Module

The mosses.mpo module provides a high-level API for Multi-Parameter Optimization analysis of compound data. It is commonly used in drug discovery to combine multiple ADMET properties into a single desirability score.

Key Features

• Sigmoid-based scoring functions for transforming raw values to 0-1 scores
• Multiple optimization algorithms for weight optimization
• ML-based weight estimation using Random Forest, Ridge, and Logistic classifiers
• Feature importance analysis via mutual information
• Enrichment and correlation statistics
• Visualization tools for analysis and comparison

Quick Start

from mosses import mpo
import pandas as pd

# Load your compound data
df = pd.read_csv("compounds.csv")

# Define parameter configurations
config = {
    "LogD": {
        "preference": "middle",      # Optimal range preferred
        "threshold": (0.0, 3.0),     # Values in this range score highest
        "weight": 1.0,
    },
    "Solubility": {
        "preference": "maximize",    # Higher is better
        "threshold": 50.0,           # Values > 50 score high
        "weight": 1.5,
    },
    "Clearance": {
        "preference": "minimize",    # Lower is better
        "threshold": 50.0,           # Values < 50 score high
        "weight": 1.0,
    },
}

# Compute MPO scores
result = mpo.compute_scores(df, config, return_intermediate=True)
print(result[["Compound Name", "MPO_Score"]].head())

Preference Types

The module supports three optimization preferences that determine how raw values are transformed into scores:

Visualizing Scoring Functions

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(-2, 6, 200)

# Maximize: values above threshold score high
maximize_scores = mpo.sigmoid(x, threshold=2.0, steepness=2.0)

# Minimize: values below threshold score high
minimize_scores = mpo.reverse_sigmoid(x, threshold=3.0, steepness=2.0)

# Middle: values in range score high
middle_scores = mpo.double_sigmoid(x, lower_threshold=1.0, upper_threshold=4.0, steepness=3.0)

Weight Optimization

Optimize weights to match experimental target data using various algorithms:

# First compute individual parameter scores
result = mpo.compute_scores(df, config, return_intermediate=True)
df_with_scores = df.merge(result, on="Compound Name")

# Define score columns
score_columns = ["LogD_score", "Solubility_score", "Clearance_score"]

# Optimize weights against experimental activity
optimized_weights, opt_result = mpo.optimize_mpo_weights(
    df_with_scores,
    score_columns,
    target_column="Activity",
    method="differential_evolution",  # or "least_squares", "minimize", "powell"
    verbose=True,
)

print("Optimized weights:", optimized_weights)

Available Optimization Methods

ML-Based Weight Estimation

Use machine learning to estimate feature importance as weights:

# Random Forest regression
ml_result = mpo.rf_regression(
    df_with_scores,
    score_columns,
    reference_col="Activity",
)

print("Feature importance:", ml_result.weights)
print(f"R² Score: {ml_result.metrics['test_r2']:.3f}")

# Other estimators available:
# mpo.rf_classifier() - Random Forest classification
# mpo.logistic_classifier() - Logistic regression
# mpo.ridge_classifier() - Ridge classifier

Evaluation Metrics

Evaluate MPO performance against experimental data:

stats = mpo.evaluate_mpo(
    df_with_scores,
    mpo_column="MPO_Score",
    reference_column="Activity",
    top_percent=10.0,  # Analyze top 10% of compounds
)

# Access metrics
print(f"Enrichment: {stats.enrichment:.2f}")
print(f"Spearman correlation: {stats.spearman_correlation:.3f}")
print(f"F1 score: {stats.f1_score:.3f}")
print(f"RMSE: {stats.rmse:.3f}")

Feature Importance Analysis

Analyze which parameters contribute most to the target:

importance_result = mpo.analyze_feature_importance(
    df_with_scores,
    score_columns,
    reference_col="Activity",
)

# Visualize
mpo.plot_mutual_info(importance_result, title="Feature Importance")

Complete Pipeline

For end-to-end analysis with automatic threshold detection and optional weight optimization:

result = mpo.build_mpo_pipeline(
    df,
    experimental_columns=["LogD", "Solubility", "Clearance", "Permeability"],
    target_column="Activity",
    preferences={
        "LogD": "middle",
        "Solubility": "maximize",
        "Clearance": "minimize",
        "Permeability": "maximize",
    },
    auto_threshold=True,  # Calculate thresholds from data
    optimize_weights_method="least_squares",  # Optional: optimize weights
)

# Result contains individual scores and final MPO_Score
print(result[["Compound Name", "MPO_Score"]].head())

Visualization Functions

The module provides several plotting utilities:

# Score distribution histogram
mpo.plot_mpo_histogram(result["MPO_Score"], title="MPO Score Distribution")

# Scatter plot with regression line
mpo.plot_best_fit_scatter(
    result["Activity"],
    result["MPO_Score"],
    label="MPO vs Activity"
)

# Correlation matrix
mpo.plot_parameter_correlation_matrix(
    df,
    columns=["LogD", "Solubility", "Clearance"],
    title="Parameter Correlations",
)

# Compare two score columns (histograms and a Venn diagram of the top compounds)
mpo.plot_comparison(
    df_with_scores,
    reference_column="Activity",
    method_column="MPO_Score",
)

API Reference

Main Functions

Scoring Functions

Statistics Functions

Plotting Functions

Example Notebook

See examples/mpo_example.ipynb for a complete walkthrough including:

1. Loading and exploring compound data
2. Configuring parameters with different preferences
3. Computing and visualizing MPO scores
4. Optimizing weights against experimental data
5. Evaluating MPO performance
6. Using ML-based weight estimation
7. Feature importance analysis
8. Building complete pipelines

License

See LICENSE.md for details.