Quality of Ersilia predictions

Quality scoring for Ersilia Model Hub predictions. eosquality describes how a model's output for a molecule compares with the same model's behaviour on a reference library and with its training sets. The scores are not ground truth: they do not estimate whether a prediction is correct. Per-column normalisation is done with eosframes.

Installation

Install the latest version of eosquality directly from GitHub:

pip install git+https://github.com/ersilia-os/eosquality.git

The CLI is then available as eosquality. Start by setting it up, which fetches the reference library:

eosquality setup

This downloads about 1.3 GB and stores the library under ~/.eosquality/indices/.

Quick start

The typical workflow is two commands: fit once per Ersilia model, then run on any query dataset against the saved artifacts. fit takes the model's predictions on the reference library (-r), its per-output-column training sets (-t), or both, and fits the matching scores. File and folder names carry the model id and version ([prefix_]<eos_id>_<version>, e.g. reference_eos4e40_v1.csv); a name without them is an error. See the CLI docs.

Fitting a reference library

The input CSV must hold the model's predictions on the exact molecules of the canonical Ersilia reference library for your installed eosquality version, in library order. It needs a key column, an input (SMILES) column, and one numeric column per model output. The artifacts folder (-a) stores the fitted scores.

There is one and only one reference library per major version of eosquality, so the molecule set is fixed by your install. If the SMILES entries in the input CSV don't match that library, fit refuses with an error.

eosquality fit -r reference_eos4e40_v1.csv -a artifacts_eos4e40_v1/
# optionally with training sets: -t training_eos4e40_v1/
# skip a score: --exclude ref_match

Please check Isaura for a large store of pre-calculations across Ersilia models.

Fitting training sets

-t takes a folder with one <output_column>.csv per model output, each with a smiles column. Given both -r and -t, the scores cover only the output columns that have a training set.

eosquality fit -r reference_eos4e40_v1.csv -t training_eos4e40_v1/ -a artifacts_eos4e40_v1/

Running against new samples

At querying time, run loads a fitted artifacts folder and scores any query CSV containing Ersilia results for the same model. A training-only artifact needs nothing but SMILES, in an input or smiles column. The output CSV has key, input, and the columns of each fitted score (see below): percentiles in (0, 1] with their *_raw values, and 1 / 0 match flags. Reference scores are named ref_…, training scores trn_….

eosquality run -i query_eos4e40_v1.csv -a artifacts_eos4e40_v1/ -o quality_eos4e40_v1.csv

Scores

Typicality and extremity compare a query against the model's own predictions on the reference library, which is not ground truth; each is calibrated so that reference molecules score roughly Uniform(0, 1). The match flags are exact lookups against the library's molecules. Training-modality scores compare it against the model's training sets instead, calibrated so that training molecules score roughly Uniform(0, 1).

The _pct columns are calibrated percentiles and the _raw columns the underlying values. run --details also writes the per-column values and the nearest training molecules.

Documentation

• Concepts: how each score is computed and calibrated
• CLI and Python API
• Architecture: fit/run data flow and save layout
• Reference library: versioning, compatibility, maintainer release steps
• Project status: current results, known limitations, open questions

License

GPL-3.0-or-later; see LICENSE.

About the Ersilia Open Source Initiative

The Ersilia Open Source Initiative is a tech-nonprofit organization fueling sustainable research in the Global South. Ersilia's main asset is the Ersilia Model Hub, an open-source repository of AI/ML models for antimicrobial drug discovery.