cpbenchy

Benchmark CPMpy solvers under proper resource limits,
change anything using plugins.

Documentation: docs.cpbenchy.com

Reliably benchmark CPMpy solvers. Every run gets its own process, its own cores and resource limits, get back one record per run: status, objective, times and memory.

From the command line or from the Python API. Interrupt it, and resume later.

Key features

• Measured properly: BenchExec's runexec with cgroups, CPU time and peak memory, pinned cores
• Any instance: XCSP3, OPB, WCNF, DIMACS, MPS and more, compressed or not, any CPMpy dataset
• Any solver: every solver CPMpy supports
• Competition-ready: follow the XCSP3 and PB competitions' rules and output
• Checked answers: every solution checked against the model
• Analysis-friendly: one record per run, access as a pandas DataFrame, get PAR-k scores
• Extensible: your own measurements, formats, ... a flexible plugin architecture

Install

pip install cpbenchy
cpbenchy doctor          # check install

Linux, Python 3.10+. For reliable limits, install BenchExec with cgroups v2; see Installation.

Quick start

# a benchmark set: the PB24 optimisation instances (or any instance files of your own)
python -c "from cpmpy.tools.datasets import OPBDataset; OPBDataset(root='data', year=2024, track='OPT-LIN', download=True)"

cpbenchy run data/opb/ -s ortools -s exact -t 60 --limit 10   # two solvers, ten instances, 60 s each
cpbenchy show                                                 # the summary, from cpbenchy-results/

Run it again without --limit, and only the missing runs get performed. Creat your first benchmark.

From Python

import cpbenchy

results = cpbenchy.run("data/opb/", solvers=["ortools", "exact"], time_limit=60)
results.to_pandas().groupby("solver").solved.sum()

For different settings per solver, results as they come in, or asyncio, see Experiments.

Make it yours

Record anything about each run with an observer, a class following the observer pattern:

class SolverTime(cpbenchy.Observer):
    def on_finish(self, ctx):                      # at the end of each run
        ctx.record("solver_time_s", ctx.solver.status().runtime)

Load your own formats, choose runs, or replace any step: see Extending cpbenchy.

What's included

• Competition rules: the limits, signals and output of the XCSP3 and PB competitions, or your own: --rules xcsp3-2025
• Competition output: answers printed as the XCSP3, PB, MaxSAT and SAT competitions expect them
• Solution checking: during the run, or afterwards on stored solutions: cpbenchy check
• PAR-k scores: rank solvers as competitions do: --par 2
• Stopping at the limit: SIGTERM at the limit, and the best solution so far: --terminate
• CPU time limits, seeds, solver parameters, cores per run, and runs in parallel
• Experiments in Python: settings per solver, results as they come in, asyncio
• Competition submissions (work in progress): a self-contained submission from rules and a solver
• From your framework (work in progress): cpbenchy measures, your framework decides and stores

The library lists everything ready to use, and the examples are there to copy.

License

cpbenchy is licensed under the Apache License 2.0.