synthonbench 0.2.0

Reference baselines, DCRF metrics and datasets for budgeted search over synthon combinatorial spaces.

synthonbench

SynthonBench is a benchmark for budgeted search over synthon combinatorial (make-on-demand) chemical spaces.

Search methods over synthon spaces never enumerate the full product library; they propose reaction-component tuples (reaction_id, synthon_ids) and pay a fixed oracle-call budget. SynthonBench compares such methods fairly and reproducibly:

• a frozen space + oracle so every run sees the same objective;
• a shared budget with a unique-query guard (duplicate oracle calls are never charged);
• the top-k DCRF headline metric — how many random oracle calls a run's top-k recall is worth, divided by what it actually spent. Random search ≈ 1.0; the exact oracle gives the ceiling.

This package ships the synthon-space task API, the frozen scalar oracles, the reference baselines, the top-k DCRF metric, the camera-ready method presets, and a one-line downloader for the released giga-scale datasets.

Install — dependency tiers

The core runs every native baseline on the synthetic oracle with NumPy only. Add tiers for parquet score tables, dataset downloads, or surrogate training:

pip install synthonbench                # core (NumPy only)
pip install "synthonbench[minimal]"     # explicit alias for the core install
pip install "synthonbench[ml]"          # + scikit-learn  → RF / ISE learned baselines
pip install "synthonbench[chem]"        # + RDKit/pandas/pyarrow → read parquet tables, build descriptors/FPs
pip install "synthonbench[hf]"          # + huggingface_hub → download the released datasets (with progress)
pip install "synthonbench[surrogate]"   # + LightGBM/XGBoost → (re)train the docking surrogates
pip install "synthonbench[cuda]"        # GPU surrogate stack (install CUDA-matched wheels yourself)
pip install "synthonbench[all]"         # everything except env-specific CUDA wheels

Quickstart (no data, no chemistry stack)

from synthonbench import make_example_task, run_method

task = make_example_task(budget=2000, seed=0)
result = run_method("ga", task)            # typed global genetic algorithm

print(result.best_utility)
print(result.metrics["call_reduction_factor_top1000"])   # top-1000 DCRF
print(result.metrics["top1000_recall"])

Compare several methods from the command line:

$ synthonbench compare --methods random_pw ts ga bb --budget 300 --seeds 0 1 2 3 4 --top-k 100
method     mean call_reduction_factor_top100  (random-equivalent baseline = 1.0)
bb         6.108
ts         3.190
ga         2.664
random_pw  1.154

$ synthonbench run --method ga --budget 2000 --seed 0
$ synthonbench list-methods

(Illustrative numbers on the bundled ~2.5k-product demo space. The paper's full-scale results are on giga-scale spaces — see below.)

Real-scale workflow (download → run → score)

pip install "synthonbench[hf,chem]"
synthonbench download --scale 1M          # spaces + glide_1M + surrogate_1M + seeds (~150 MB)

from synthonbench import load_space, load_oracle, build_with_preset
from synthonbench.runner import run_optimizer
from synthonbench.task import GlobalSynthonTask

space  = load_space("1M")
oracle = load_oracle("1M", "kif11", kind="glide")     # frozen real-Glide LE oracle
task   = GlobalSynthonTask(space, oracle, budget=10_000, seed=0)

opt    = build_with_preset("ga", space=space, allowed_reactions=task.allowed_reactions, seed=0)
result = run_optimizer(opt, task, method="ga")

A full, copy-pasteable submission is in examples/submission_template.py; score it against the released table with scripts/score_submission.py.

The baseline roster

Names, abbreviations and Family mirror the camera-ready leaderboard. Every method is a global optimizer over the shared multi-reaction budget; most are a single-reaction proposer made global by a reaction allocator, and the typed GA is global by construction.

name	label	family	what it is
ga	GA	Evolutionary	typed global genetic algorithm — strongest in-house baseline
ise	ISE	Active learning	iterative surrogate expansion (SpaceHASTEN-style, benchmark-native)
salsa	SALSA*	Bandit	SALSA-inspired tabular Thompson sampler
bb	BB*	Bandit	ROCS-X-inspired Bayesian belief bandit
ts	TS	Bandit	reagent-prior Thompson sampling (Klarich/Walters-style prior)
sh	SpaceHASTEN	Similarity search	external — FTrees/SpaceLight + surrogate
rf	RF	Active learning	pooled per-slot random forest (needs [ml])
shc	SpaceHASTEN-ctrl	Similarity search	external — SpaceHASTEN without the surrogate
random_rw	Rand-RW	Random	reaction-uniform random anchor
mcts	MCTS	Metaheuristic	Monte Carlo tree search made global
hc	HC	Metaheuristic	hill-climbing made global
random_pw	Rand-PW	Random	product-weighted random anchor (DCRF ≈ 1 cost baseline)

Appendix / reference rows: spacega, hasten, molpal, ts_walters (all external), and the registered ablations fm, thompson_slot, slot_ucb. synthonbench list-methods prints the full roster with families.

Starred (*) rows are benchmark-native analogues inspired by the named systems' public descriptions, not source-faithful reproductions. ise is a SpaceHASTEN-style analogue, not the SpaceHASTEN source adapter.

External methods (optional, not bundled)

sh, shc, spacega, hasten, molpal and ts_walters are third-party or licensed systems the paper drove through their own upstream code. They are in the roster for completeness and result reproduction; running them means cloning the upstream repo (and, for some, commercial software). Nothing is bundled.

synthonbench fetch                 # list the optional backends + their licenses/URLs
synthonbench fetch spacega         # git-clone one into ~/.cache/synthonbench/repos

key	upstream	license	also needs
spacehasten (sh,shc)	TuomoKalliokoski/SpaceHASTEN	Proprietary (Orion)	Glide, BiosolveIT FTrees/CoLibri
hasten	TuomoKalliokoski/HASTEN	upstream	Glide, ChemProp/XGBoost
spacega	lmoesgaard/SpaceGA	MIT	BiosolveIT SpaceLight
thompson_sampling	PatWalters/TS	MIT	RDKit
molpal	coleygroup/molpal	MIT	RDKit, PyTorch
rdkit_fp	mireklzicar/rdkit-fp (pip install rdfp)	BSD-3	RDKit

create_optimizer("sh", ...) raises a clear, actionable ExternalMethodError with the fetch + install instructions rather than silently failing.

Datasets (Hugging Face)

Frozen tasks, score tables, synthon spaces and shared warm-start seeds live in mireklzicar/synthonbench.

synthonbench download --scale 1M                     # one scale
synthonbench download --scale all                    # incl. the 100M table (~5 GB)
synthonbench download --scale 10M --what scores      # just the 10M surrogate table

group	files	notes
spaces/	synthon_space_{1M,10M,100M}.synthons.tsv + properties + reactions.tsv	42 canonical reactions (SMIRKS)
scores/glide_1M.parquet	real-Glide LE oracle, 3 targets	headline docking track (medchem-filtered, −Glide−cLogP)
scores/surrogate_{1M,10M,100M}.parquet	LightGBM surrogate, 3 targets	scale tracks
seeds/seeds_<target>_s{0,1,2}.parquet	3 × 100k shared warm-start pools / target	one per protocol seed

Targets: KIF11, PYRD, TGFR1. Scales: 990,610 / 10,022,100 / 99,851,700 products. All tables key on the same md5 product_id; the surrogate and seed tables also carry the synthon{1,2,3}_id tuple.

The DCRF metric

The headline efficiency metric is the top-k DCRF:

recall@k = |observed ∩ exact_top_k| / k
DCRF_k   = recall@k / (B / N)          # = recall@k * N / B

where B is the charged (unique) oracle-call budget and N the number of products in the (allowed) space. Random search has DCRF = 1 in expectation; the exact oracle gives the ceiling. The paper audits the scale-normalized top-0.1%: top-1000 @ 1M, top-10,000 @ 10M, top-100,000 @ 100M.

• For small spaces, synthonbench.metrics.compute_exact_audit enumerates and ranks the space exactly (used by run_method).
• At 1M/10M/100M, scripts/score_submission.py ranks the released score table directly — no enumeration — and scores a submission's recall@k / DCRF@k.

Method presets & curricula

synthonbench.presets carries the exact camera-ready hyperparameters (the appendix table) mapped onto the optimizer constructors, plus the protocol constants:

from synthonbench import preset_kwargs, build_with_preset
from synthonbench.presets import BUDGET_SWEEP, SEEDS, TOPK_BY_SCALE

preset_kwargs("ga")     # {'population_size': 96, 'mutate_p': 0.35, ...}
BUDGET_SWEEP            # (250, 500, 1000, 2500, 5000, 10000, 100000)
SEEDS                  # (0, 1, 2, 3, 4)   — protocol minimum is (0, 1, 2)
TOPK_BY_SCALE          # {'1M': 1000, '10M': 10000, '100M': 100000}

Reproduction scripts

See scripts/:

• scripts/score_submission.py — at-scale DCRF scorer.
• scripts/fingerprints/compute_descriptors.py — RDKit physchem descriptors + ECFP4 fingerprints via rdfp.
• scripts/colibri/ — build BioSolveIT CoLibri/infiniSee .space files from the synthon TSVs (for the SpaceHASTEN/SpaceGA/FTrees backends; requires a CoLibri license).

Practical notes — runtime & memory

Native baselines are pure-Python ask/tell loops; cost scales with the budget (each oracle call is an O(1) table lookup), not with the space size. Order-of-magnitude guidance on one modern core:

scale	score table on disk	RAM to load/score	notes
1M	~47 MB (surrogate) / ~93 MB (glide)	< 1 GB	frozen oracle dict builds in seconds
10M	~480 MB	~2–4 GB	read only the columns you need
100M	~4.6 GB	~16–32 GB	load the product_id + one score column only; this is the RAM wall

A 10,000-budget native run on the 1M oracle finishes in seconds to a couple of minutes; [ml]/surrogate methods (RF/ISE) are dominated by periodic refits. External backends (Glide docking, FTrees) are orders of magnitude heavier and governed by their own tooling.

Development

git clone https://github.com/mireklzicar/synthonbench && cd synthonbench
pip install -e ".[dev]"
ruff check src tests
pytest -q
python -m build

Status

The camera-ready paper describes the benchmark design, tasks, oracles and full baseline results. This repository is the public home for the package, the reference baselines, the presets, and the reproduction scripts; the datasets are on the Hub.

Links

• Project / demo: https://synthonbench.mireklzicar.com/
• Dataset: https://huggingface.co/datasets/mireklzicar/synthonbench
• Companion packages: SynthonOR (synthon similarity search), SynthonMap (embedding + density rendering), rdkit-fp (rdfp, fingerprints).

Citation

If you use SynthonBench, please cite it using the metadata in CITATION.cff.

License

Released under the MIT License. External backends and datasets carry their own licenses (see above).