CatBench

A benchmarking framework for Machine Learning Interatomic Potentials in heterogeneous catalysis.

CatBench evaluates MLIPs against DFT references across four task types — adsorption energy, surface energy, bulk formation energy, and equation of state — with automated data processing, reproducible calculation workflows, and statistical anomaly detection.

Table of Contents

• Installation
• Quick Start
• Tutorials
• Adsorption Energy Benchmarking
  • Data Preparation
  • Calculation
  • Analysis
  • Output Files
• Relative Energy Benchmarking
  • Surface Energy
  • Bulk Formation Energy
• Equation of State (EOS) Benchmarking
• Configuration Reference
• Citation

Installation

# Basic
pip install catbench

# With D3 dispersion correction (GPU required; CPU-only not currently supported)
pip install catbench[d3]

# Development install from source
git clone https://github.com/JinukMoon/CatBench.git
cd CatBench
pip install -e .

Quick Start

Minimum viable benchmark in 5 lines:

from catbench.adsorption import get_benchmark, AdsorptionCalculation, AdsorptionAnalysis
from your_mlip import YourCalculator

get_benchmark("BM_dataset")                                              # 0.3 MB
calc = YourCalculator(...)
AdsorptionCalculation([calc] * 3, mlip_name="MyMLIP", benchmark="BM_dataset").run()
AdsorptionAnalysis().analysis()                                          # parity plots + Excel

For an end-to-end walkthrough on the publication's main benchmark (MamunHighT2019) with MACE-MP-0, see the tutorial notebook.

Tutorials

tutorials/catbench_tutorial.ipynb — complete walkthrough on a 20-reaction subset of MamunHighT2019 using MACE-MP-0 as the representative MLIP. Covers Zenodo download, calculator setup, calculation, analysis, and the multi-MLIP comparison workflow. Runs in about 10 minutes on a Colab T4 GPU.

Adsorption Energy Benchmarking

Data Preparation

Option A: get_benchmark (Recommended)

get_benchmark(name) fetches a dataset from the best available source automatically, so you can request any benchmark by name without knowing where it lives:

from catbench.adsorption import get_benchmark, list_zenodo_benchmarks

get_benchmark("MamunHighT2019")   # writes raw_data/MamunHighT2019_adsorption.json

It resolves each name through three sources in order:

1. Zenodo — the featured datasets below, citable with a DOI (concept DOI: 10.5281/zenodo.17157085, always resolves to the latest version).
2. CatBench leaderboard (catbench.org) — every dataset on the leaderboard, gzip-compressed for a fast download.
3. CatHub — direct download + preprocessing for anything not hosted above.

Featured datasets (hosted on Zenodo):

list_zenodo_benchmarks()   # the featured names currently on Zenodo

For explicit control, the source-specific functions still work directly: zenodo_download(name) (Zenodo only) and cathub_preprocessing(name).

If catbench.org cannot be reached (network error), get_benchmark stops with an error instead of silently falling back to CatHub (1.1.5).

Option B: CatHub Database

cathub_preprocessing(name) takes the preprocessed file from the catbench.org mirror when it has the dataset (no key needed) and downloads from CatHub only otherwise (source="auto", default). source="mirror" uses the mirror only, source="cathub" always downloads from CatHub.

CatHub API key (required since 2026). CatHub now answers only requests that carry a personal API key. Get one at https://api-catalysis-hub.slac.stanford.edu/auth/login, then either

export CATHUB_API_KEY=<your key>

or pass it explicitly (never commit it to a script):

import os
from catbench.adsorption import cathub_preprocessing

cathub_preprocessing("NewDataset2026", source="cathub", api_key=os.environ["CATHUB_API_KEY"])

CatHub request limits. CatHub allows 10 requests/minute and suspends accounts above 500 requests/day, and it returns structures one reaction per request. CatBench spaces requests 6.5 s apart and refuses, before downloading, any dataset that would need more requests than request_budget (default 300). Large datasets such as MamunHighT2019 (88,587 CatHub reactions) cannot be downloaded from CatHub in one day — use the mirror.

Downloads are deterministic (stable order: "id" pagination + id-based dedup) and fixed-atom constraints are handled automatically: CatHub's deposited constraints are kept as-is, and for datasets where CatHub omits them the fixed set is inferred from geometry (clean slab vs adslab — atoms that do not move were held fixed) and injected as FixAtoms. A genuinely unconstrained slab is left free. So every preprocessed dataset is self-describing and runs correctly out of the box.

Combining datasets and unifying adsorbate names. Pass a list of benchmarks to process them together, and an optional adsorbate_integration map to merge inconsistent adsorbate labels (e.g. HO → OH) into a single name:

cathub_preprocessing(
    ["MamunHighT2019", "AraComputational2022"],
    adsorbate_integration={"HO": "OH", "O2H": "OOH"},
)

Option C: User VASP Data

Note: Fixed-atom constraints are taken from the data automatically — your VASP Selective-Dynamics (T/F) flags are read into FixAtoms on the CONTCAR and applied during relaxation.

Warning: vasp_preprocessing deletes every file except CONTCAR and OSZICAR to save disk space. Always run it on a copy of your original VASP output.

Organize the data as follows. gas, slab, and the <name>gas pattern are reserved; everything else is arbitrary:

your_dataset_name/
├── gas/
│   ├── H2gas/     { CONTCAR, OSZICAR }    # gas-reference folder, must end with "gas"
│   └── H2Ogas/    { CONTCAR, OSZICAR }
├── system_A/
│   ├── slab/      { CONTCAR, OSZICAR }    # reserved name (clean surface)
│   ├── H/
│   │   ├── site_0/  { CONTCAR, OSZICAR }  # any name for site variants
│   │   └── site_1/  { CONTCAR, OSZICAR }
│   └── OH/
│       └── ...
└── system_B/
    └── ...

Declare the reaction stoichiometry and preprocess:

from catbench.adsorption import vasp_preprocessing

coeff_setting = {
    "H":  {"slab": -1, "adslab": 1, "H2gas": -1/2},
    "OH": {"slab": -1, "adslab": 1, "H2gas": +1/2, "H2Ogas": -1},
}
vasp_preprocessing("your_dataset_name", coeff_setting)
# → raw_data/your_dataset_name_adsorption.json

The keys "slab" and "adslab" are required literals on every entry; all other keys are gas-phase references and must end with "gas". vasp_preprocessing validates these rules before deleting anything.

Calculation

AdsorptionCalculation takes a list of calculators. Running the same calculator multiple times provides reproducibility statistics that the analysis step uses for anomaly detection.

from catbench.adsorption import AdsorptionCalculation
from your_mlip import YourCalculator

calc = YourCalculator(...)

AdsorptionCalculation(
    [calc] * 3,                    # 3 reproducibility seeds
    mlip_name="YourMLIP",          # free-form label — folder name under result/, display name in plots
    benchmark="dataset_name",
    # save_files=False,            # skip trajectory + log files to save disk space
).run()

D3 dispersion correction (GPU required):

from catbench.dispersion import DispersionCorrection

d3 = DispersionCorrection()                     # Becke-Johnson damping + PBE by default
calc_d3 = d3.apply(YourCalculator(...))
AdsorptionCalculation([calc_d3] * 3, mlip_name="YourMLIP_D3", benchmark="dataset_name").run()

OC20 mode for MLIPs that predict adsorption energy directly:

AdsorptionCalculation(
    [oc20_calc] * 3, mode="oc20", mlip_name="OC20_MLIP", benchmark="dataset_name",
).run()

See the Configuration Reference for all options, and the tutorial notebook for an end-to-end runnable example.

Analysis

from catbench.adsorption import AdsorptionAnalysis

AdsorptionAnalysis().analysis()    # auto-detects every MLIP under ./result/

This produces:

• Parity plots under ./plot/<mlip_name>/{mono,multi}/ — mono/total.png aggregates all reactions; multi/total.png colors by adsorbate.
• Excel report ./{cwd_name}_Benchmarking_Analysis.xlsx with MAE, RMSE, anomaly breakdown, ADwT, AMDwT, and timings across every MLIP in ./result/.

Every data point is classified into Normal, Migration, Energy Anom., Unphys. Relax, or Reprod. Fail. Thresholds are configurable — see the Configuration Reference.

Threshold sensitivity:

AdsorptionAnalysis().threshold_sensitivity_analysis()   # displacement + bond-length by default

This generates stacked-area charts showing how anomaly-classification rates change with threshold values.

Gas-reference shift correction (v1.1.4+): every Excel report also carries a gas-shift-corrected view. For each adsorbate, the median signed error (MLIP − DFT) over the structure-valid reactions (normal + energy anomaly; structural failures excluded) is removed as a constant shift — a systematic gas-reference offset correction (the median, unlike the mean, is not dragged by a minority cluster of wild outliers) — and the energy-anomaly classification is re-evaluated on the shifted errors. Groups with fewer than gas_shift_min_n (default 5) structure-valid reactions are left uncorrected. Results land in the MLIP_Data_shifted / anomaly_shifted twin sheets and a per-adsorbate "Gas shift correction" block (applied shift, N_fit, shifted MAEs) on each MLIP sheet; the original sheets are unchanged. When plotting is enabled, gas-shifted parity plots are also drawn (multi/total_shifted.png, multi/normal_shifted.png). The leaderboard exposes the same correction as a "Gas shift" toggle.

Output Files

Parity plots

Mono — all reactions combined	Multi — colored by adsorbate

Excel report

The Excel workbook has three sheet types. Example numbers from the paper:

Main comparison sheet — one row per MLIP:

Threshold sensitivity charts

Displacement threshold	Bond-length change threshold

Relative Energy Benchmarking

Same data → calculation → analysis shape as Adsorption, but with a single calculator (no reproducibility seeds) and a task_type dispatch.

Surface Energy

Warning: Preprocessing deletes all files except CONTCAR and OSZICAR. Always work on a copy.

Layout — per material, one bulk/ and one slab/:

your_surface_data/
├── Material_1/
│   ├── bulk/  { CONTCAR, OSZICAR }
│   └── slab/  { CONTCAR, OSZICAR }
├── Material_2/
│   └── ...

from catbench.relative.surface_energy.data import surface_energy_vasp_preprocessing
from catbench.relative import SurfaceEnergyCalculation, RelativeEnergyAnalysis

surface_energy_vasp_preprocessing("your_surface_data")
SurfaceEnergyCalculation(calculator=calc, benchmark="your_surface_data", mlip_name="MyMLIP").run()
RelativeEnergyAnalysis(task_type="surface").analysis()

The Excel report provides MAE, RMSE, and max error (J/m2) across all surfaces per MLIP.

Bulk Formation Energy

Warning: Preprocessing deletes all files except CONTCAR and OSZICAR.

Layout — bulk_compounds/ and elements/ side-by-side:

your_formation_data/
├── bulk_compounds/
│   ├── Compound_1/  { CONTCAR, OSZICAR }
│   └── Compound_2/  { CONTCAR, OSZICAR }
└── elements/
    ├── Element_A/   { CONTCAR, OSZICAR }
    ├── Element_B/   { CONTCAR, OSZICAR }
    └── Element_C/   { CONTCAR, OSZICAR }

from catbench.relative.bulk_formation.data import bulk_formation_vasp_preprocessing
from catbench.relative import BulkFormationCalculation, RelativeEnergyAnalysis

coeff_setting = {
    "Compound_1": {"bulk": 1, "Element_A": -1, "Element_C": -1/2},
    "Compound_2": {"bulk": 1, "Element_B": -2, "Element_C": -3/2},
}
bulk_formation_vasp_preprocessing("your_formation_data", coeff_setting)
BulkFormationCalculation(calculator=calc, benchmark="your_formation_data", mlip_name="MyMLIP").run()
RelativeEnergyAnalysis(task_type="bulk_formation").analysis()

Equation of State (EOS) Benchmarking

Each material has N volume-point subfolders named 0, 1, …, N:

your_eos_data/
├── Material_1/
│   ├── 0/   { CONTCAR, OSZICAR }    # smallest volume
│   ├── 1/   { CONTCAR, OSZICAR }
│   └── ...   (up to 10, typically)
├── Material_2/
│   └── ...

from catbench.eos import eos_vasp_preprocessing, EOSCalculation, EOSAnalysis

eos_vasp_preprocessing("your_eos_data")
EOSCalculation(calculator=calc, mlip_name="MyMLIP", benchmark="your_eos_data").run()
EOSAnalysis().analysis()

The Excel report includes Birch-Murnaghan fits with bulk modulus (B0), equilibrium volume (V0), and derivative (B0'):

Configuration Reference

Options are grouped into Required, Commonly tuned, and Advanced (collapsed). Required parameters must be passed at construction; the rest have sensible defaults and can be overridden as needed.

AdsorptionCalculation

Required

Commonly tuned

AdsorptionAnalysis

Commonly tuned

DispersionCorrection

Relative energy and EOS classes

SurfaceEnergyCalculation, BulkFormationCalculation, and EOSCalculation all require calculator, benchmark, and mlip_name, and accept f_crit_relax and n_crit_relax for optimization control.

RelativeEnergyAnalysis requires task_type ("surface", "bulk_formation", or "custom") and accepts the same plotting options as AdsorptionAnalysis.

Citation

@article{catbench2025,
  title={CatBench Framework for Benchmarking Machine Learning Interatomic Potentials in Adsorption Energy Predictions for Heterogeneous Catalysis},
  author={Moon, Jinuk and Jeon, Uchan and Choung, Seokhyun and Han, Jeong Woo},
  journal={Cell Reports Physical Science},
  volume={6},
  pages={102968},
  year={2025},
  doi={10.1016/j.xcrp.2025.102968}
}

License

MIT — see LICENSE.

Contact

Jinuk Moon · jumoon@snu.ac.kr
Jeong Woo Han · jwhan98@snu.ac.kr
Seoul National University

For bug reports, feature requests, and contributions: GitHub repository.