3SAlgorythm

Bayesian optimization benchmark: is it worth diversifying the batch when picking which compounds to test in each round of a screen, instead of simply grabbing whatever the model scores highest?

Executive summary with plots, molecules, and findings: docs/FINDINGS.md (readable by anyone with access to this repo, no claude.ai login needed; there's also an interactive version, only visible if you're signed into the account that published it).

What's here

divbatch/ is an installable package with the reusable core; everything else at the repo root is the benchmark harness and scripts that use it.

• divbatch/objectives.py — objective functions: CompoundScreen (synthetic screen), Branin, Hartmann6, and RealCompoundScreen (real chemistry — any SMILES + continuous-property CSV, MoleculeNet-style; ECFP4 fingerprints via RDKit).
• divbatch/optimizers.py — RandomPicker, GPExpectedImprovement with pluggable batch selection (TopKSelector, LocalPenalizationSelector, DPPSelector, KrigingBelieverSelector) and a Tanimoto kernel for real chemistry. Core package API — only needs numpy/scipy/scikit-learn.
• divbatch/recommend.py — divbatch-recommend, a CLI that runs the actual TopK/LocalPenalization/DPP comparison on your own SMILES+activity CSV and reports which one performed best on your data. Exists because Experiment 7 found no cheap, universal rule for which selector wins on a given library — so instead of guessing, this measures it directly, at a modest default budget. Installed automatically with the package.
• divbatch/optimizers_torch.py — BoTorch/GPyTorch optimizers (optional, needs the torch extra): BoTorchQEI (off-the-shelf qLogExpectedImprovement, jointly optimized over the batch); TanimotoLocalPenalizedQEI (hybrid — BoTorch's fitted GP for the quality signal, LocalPenalization's quality-agnostic repulsion for diversity, repulsion computed from the GP's own Tanimoto kernel instead of an ad-hoc Lipschitz radius); SparseTanimotoLocalPenalizedGP (same mechanism on a stochastic variational GP — scales to training sets an exact GP can't afford).
• scalability_demo.py — exact vs. sparse GP fit time as the training set grows past what this project's real dataset can exercise (synthetic fingerprints, since BACE only has 1513 compounds).
• generalization_test.py — repeats the TopK/LocalPenalization/DPP comparison on real MoleculeNet regression datasets (drug-like and quantum-chemistry) to check whether the BACE finding generalizes. It mostly doesn't — see docs/FINDINGS.md, Experiment 7.
• isolation_diagnostic.py — attempts to predict, from fingerprint geometry alone (no optimizer runs), which datasets LocalPenalization will win on. Negative result, confirmed by extending the dataset pool from 10 to 30 — the weak correlations at n=10 got weaker and partly flipped sign at n=30. See docs/FINDINGS.md.
• benchmark.py — command-line runner: sweeps optimizer × batch size × seed, saves results to JSON, and produces a comparison plot.
• test_benchmark.py — regression tests (pytest), also run on every push via GitHub Actions.
• ALGORITHMS.md / SPEC.md — the original spec and the mathematical detail of each selector.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,chem]"
# optional, only for divbatch/optimizers_torch.py:
pip install -e ".[torch]"

chem pulls in RDKit (needed for RealCompoundScreen); dev pulls in pytest and matplotlib (needed for the benchmark/test scripts, not for the divbatch package itself). Someone who only wants divbatch's core selectors in their own project can just pip install 3SAlgorythm (once published) or pip install -e . here with no extras — numpy/scipy/ scikit-learn are the only hard dependencies. The distribution is named 3SAlgorythm (matches this repo), but the importable package is still divbatch — pip install 3SAlgorythm then import divbatch in your code, the same split as e.g. pip install beautifulsoup4 / import bs4.

The real-chemistry scenario uses MoleculeNet's BACE dataset:

mkdir -p data
curl -o data/bace.csv https://deepchemdata.s3-us-west-1.amazonaws.com/datasets/bace.csv

generalization_test.py (Experiment 7) needs 6 more MoleculeNet CSVs, same source; --extra (the 20-dataset isolation-diagnostic follow-up) needs 2 more on top of those:

for f in delaney-processed.csv SAMPL.csv Lipophilicity.csv clearance.csv qm7.csv qm8.csv qm9.csv thermosol.csv hppb.csv; do
  curl -o "data/$f" "https://deepchemdata.s3-us-west-1.amazonaws.com/datasets/$f"
done

Usage

pytest test_benchmark.py -q

python benchmark.py --quick                                    # quick sweep, synthetic
python benchmark.py                                             # full sweep (10 seeds, synthetic)
python benchmark.py --dataset real --seeds 0 1 2 3 4             # real data, PCA+Matern kernel
python benchmark.py --dataset real --optimizers GP-EI+LocalPenalization+Tanimoto \
    --seeds 0 1 2 3 4 --batch-sizes 25 50 --lp-alpha 1.0         # the study's winner

--lp-alpha controls LocalPenalizationSelector's "risk appetite": 0 ignores quality and prioritizes pure spatial novelty, 1 is the original behavior, higher values are more conservative.

divbatch-recommend — which selector should you use?

This project's own science (Experiment 7, docs/FINDINGS.md) found no reliable, cheap-to-compute rule for predicting whether LocalPenalizationSelector will beat plain top-EI selection on a given library — it depends on properties of the data that isolation/clustering metrics didn't capture, even across 30 independent test datasets. So rather than ship a rule, divbatch-recommend runs the actual comparison on your data:

pip install "3SAlgorythm[chem]"
divbatch-recommend my_screen.csv --smiles-col smiles --target-col activity

It needs a CSV with a SMILES column and a continuous property to maximize (any MoleculeNet-style layout works). By default it simulates a 100-compound budget in batches of 25, repeated over 6 random seeds — fast enough to run interactively; --seeds, --batch-size, and --n-experiments scale it up for a more reliable answer at the cost of runtime. It also flags a specific failure mode this project ran into (Experiment 7's Clearance dataset): a target capped at an assay ceiling, where many compounds tie for the maximum and any strategy "wins" trivially.

Where this fits

A quick survey of what's already out there, and why this project isn't redundant with it:

• BoTorch/Ax (Meta) is the strongest general-purpose framework and what divbatch/optimizers_torch.py is built on. Its off-the-shelf batch acquisition (qLogExpectedImprovement) is joint-correct — it avoids picking near-duplicates by construction — but it's still quality-weighted, so it never bets on a genuinely low-predicted-improvement structural outlier. Confirmed twice in this project: with our own Tanimoto kernel, and independently with GAUCHE's (see below) — same result both times.
• GAUCHE ships Tanimoto/fingerprint kernels for BoTorch out of the box — essentially a packaged version of divbatch.optimizers.TanimotoKernel / divbatch.optimizers_torch.TanimotoKernelTorch. It doesn't ship a quality-agnostic batch selector, so it inherits the same blind spot as plain BoTorch unless you add one.
• Optuna, the most popular hyperparameter-tuning library, distributes parallel workers close to independently — the same redundancy problem TopKSelector has, largely unaddressed.
• EDBO targets reaction optimization (catalyst × solvent × temperature), not library screening — a different problem shape than this project's.
• Google Vertex AI Vizier, AWS SageMaker Automatic Model Tuning — managed, black-box, built for ML hyperparameter tuning rather than experiment design with a domain-specific kernel.
• SigOpt (Intel) was discontinued as a managed service in September 2023; its code lives on as the unmaintained sigopt-server.

None of the above ships the mechanism this project's main finding depends on — diversity that ignores predicted quality once something is spatially unique — as a reusable component. TanimotoLocalPenalizedQEI in divbatch/optimizers_torch.py is the attempt at closing that gap: BoTorch's own fitted GP for the quality signal, LocalPenalization's repulsion for diversity, computed from the GP's real Tanimoto kernel instead of a separately-estimated Lipschitz constant.

Validated over 30 seeds on BACE: it finds the same structural outlier as the hand-rolled sklearn selector in 27/30 (q=25) and 26/30 (q=50) seeds — against 0/30 for plain qLogExpectedImprovement and 0/30 for DPP — while running at 0.02–0.03s/round, faster than the sklearn version (0.2s) and ~250x faster than off-the-shelf qLogExpectedImprovement (5–7s) at this scale.

Main finding

On real data (BACE, 1513 compounds), with a Tanimoto kernel instead of the usual PCA projection, LocalPenalizationSelector finds the most potent compound in the entire library across all 30 seeds tested, without exception — neither DPP, nor KrigingBeliever, nor BoTorch's native acquisition function (qLogExpectedImprovement) manage it. Independently confirmed with GAUCHE's own Tanimoto kernel implementation, not just ours.

But it doesn't generalize. Repeating the same comparison on 9 more real datasets (generalization_test.py) gives 2 clear wins, 2 clear losses, and 6 ties out of 10 total — no domain pattern, roughly a coin flip. The BACE result depends on a genuine structural outlier existing in the library; when there isn't one, LocalPenalization's quality-agnostic exploration is a mild liability, not a free win. This is not a safe default upgrade over TopK — see docs/FINDINGS.md, Experiment 7, for the full breakdown and the honest recommendation table.

Publishing (for maintainers)

Not yet published to PyPI. .github/workflows/publish.yml builds the sdist and wheel and publishes on every GitHub Release, via PyPI Trusted Publishing (OIDC — no API token stored in the repo). PyPI project name is 3SAlgorythm (the distribution name in pyproject.toml; the importable package stays divbatch — see Installation above), with a pending publisher already registered for jmiguelmangas/3SAlgorythm, workflow publish.yml. After that, publishing is just cutting a GitHub Release.

python -m build && twine check dist/* passes locally as of this package's current state — the metadata and both artifacts are valid.

License

MIT.