qpsolvers_benchmark 0.1.0rc1

Benchmark for quadratic programming solvers available in Python.

QP solvers benchmark

Benchmark for quadratic programming (QP) solvers available in Python. It contains the following test sets:

• Maros-Meszaros: problems designed to be difficult, some of them with non-strictly p.s.d. Hessians.
• Maros-Meszaros dense: subset of the 59 smallest Maros-Meszaros problems, still difficult but accessible to dense solvers.

This benchmark aims to help us compare and select QP solvers. Its methodology is open to discussions. New test sets are also welcome. Feel free to add one that better represents the family of problems you are working on.

Solvers

Solver	Keyword	Algorithm	Matrices	License
CVXOPT	cvxopt	Interior point	Dense	GPL-3.0
ECOS	ecos	Interior point	Sparse	GPL-3.0
Gurobi	gurobi	Interior point	Sparse	Commercial
HiGHS	highs	Active set	Sparse	MIT
MOSEK	mosek	Interior point	Sparse	Commercial
OSQP	osqp	Augmented Lagrangian	Sparse	Apache-2.0
ProxQP	proxqp	Augmented Lagrangian	Dense & Sparse	BSD-2-Clause
qpOASES	qpoases	Active set	Dense	LGPL-2.1
qpSWIFT	qpswift	Interior point	Sparse	GPL-3.0
quadprog	quadprog	Active set	Dense	GPL-2.0
SCS	scs	Augmented Lagrangian	Sparse	MIT

Results

Check out the full reports for each test set in the results directory.

Maros-Meszaros

The Maros-Meszaros test set contains difficult problems, some of them large, sparse, ill-conditioned or with non-strictly p.s.d. Hessian matrices. For sparse solvers only.

Solver	Success rate (%)	Runtime (× slower than best)	Primal error
cvxopt	16	16.4	18.5
highs	61	1.8	2.0
osqp	64	1.3	1.8
proxqp	72	1.0	1.0
scs	54	3.1	548.2

Check out the full report for definitions and details.

Maros-Meszaros dense

Probems in the Maros-Meszaros test set with less than a 1,000 optimization variables and 1,000 constraints. Note that this subset is not representative of the full Maros-Meszaros test set.

Check out the full report for definitions and details.

Limitations

Here are some known areas of improvement for this benchmark:

• Cold start only: we don't evaluate warm-start performance for now.
• Dual feasibility: we don't check the dual multipliers that solvers compute internally, as the API for them is not yet unified.

Running the benchmark

To run the benchmark on your machine, you will first need to install qpsolvers along with the QP solvers you want to test. For instance, to install open source solvers:

pip install qpsolvers[open_source_solvers]

To run e.g. the Maros-Meszaros test set, call:

python benchmark.py run maros_meszaros

Replace maros_meszaros with the name of the test set you want to run. You can also run only a specified solver, problem or set of solver settings, for instance:

python benchmark.py run maros_meszaros_dense --solver proxqp --settings default

Check out python benchmark.py --help for details.

See also

• Optimality conditions and numerical tolerances in QP solvers: note written while figuring out the high_accuracy settings of this benchmark.
• jrl-qp/benchmarks: benchmark of QP solvers available in C++.
• proxqp_benchmark: benchmark examples for the ProxQP solver.