Skip to main content

Quality Profiles

Benchmark optimization solvers by the quality of the solutions they return.

Quality Profiles: benchmark optimization solvers by the quality of their solutions

Docs Paper License: MIT GitHub Stars
Tests Lint Ruff Python Platform
PyPI version PyPI Downloads Conda version Conda Downloads

Installation · Quick start · Plot styles · What is computed · API · Paper's experiments · Documentation · Citation · Contributing


Performance and data profiles measure how much effort a solver needs. Quality profiles measure something else: how good the final objective values are, relative to a common starting point and a reference value, over the whole range of precision levels. They need no a priori thresholds, and they handle failures and solvers that converge to different local minimizers, so no test problem has to be excluded.

This package implements the method of

G. Fasano, C. Piermarini, M. Roma. Benchmarking Optimization Algorithms with Quality Profiles and Test Set Profiles. Mathematical Programming Computation (accepted). arXiv:2607.16798

It is a Python port of the authors' reference MATLAB code (fasano-g/Quality-Profiles). On the same grid it gives exactly the same values, and it runs much faster: 20,000 problems and 20 solvers take a few hundredths of a second.

Quality profiles of L-BFGS, CG+, TRON and CG_DESCENT with r1 = 1 and r1 = 5
Four solvers on 166 CUTEst problems (Figures 4 and 5 of the paper). With r1 = 5 the plot reaches 18 orders of magnitude and shows that TRON stays ahead at high accuracy.

Installation

Quality Profiles is published on PyPI as quality-profiles and imported in Python as quality_profiles. From PyPI:

pip install quality-profiles                    # NumPy and Matplotlib only
pip install "quality-profiles[science]"         # + SciencePlots styles, for figures in papers
pip install "quality-profiles[interactive]"     # + Plotly, for interactive exploration
pip install "quality-profiles[all]"             # everything

The core package depends only on NumPy and Matplotlib. The extras are kept separate so that a solver's test suite can use quality profiles without pulling in Plotly.

From source (development), with uv:

git clone https://github.com/chripiermarini/QualityProfiles
cd QualityProfiles
uv sync                    # .venv with the package, Plotly, SciencePlots and the dev tools

Requires Python 3.10 or later.

Quality Profiles follows semantic versioning: within version 1.x the public API does not change in incompatible ways. See the changelog.

Quick start

For every test problem p you need the objective value at the common starting point, f0[p], and the best value found by each solver, f_solvers[p, s]. Use NaN to mark a failure.

import numpy as np
import matplotlib.pyplot as plt
from quality_profiles import quality_profiles, plot_quality_profiles

f0 = np.array([10.0, 3.0, 50.0, 7.0])           # f(x0) on 4 problems
f_solvers = np.array([                           # 3 solvers
    [1e-8,  2e-3,  0.5],
    [0.0,   1e-6,  1e-12],
    [12.0,  1e-4,  np.nan],                      # nan = failure
    [1e-3,  1e-3,  1e-9],
])

qp = quality_profiles(f0, f_solvers, solver_names=["A", "B", "C"], r1=4)

plot_quality_profiles(qp)      # semilog plot of [Q_s(tau^r1)]^r2
plt.show()

qp.evaluate([1e-1, 1e-3, 1e-6])  # fraction solved to 1, 3 and 6 orders of magnitude
qp.ranking()                     # solvers sorted by area under the profile

For a figure in a paper, or for interactive exploration:

ax = plot_quality_profiles(qp, style=["science", "ieee"])  # needs the [science] extra
ax.figure.savefig("profiles.pdf")

from quality_profiles import plot_quality_profiles_interactive
fig = plot_quality_profiles_interactive(qp)                # needs the [interactive] extra
fig.show()                                                 # zoom, hover, hide solvers
fig.write_html("profiles.html")

The notebook notebooks/quality_profiles_example.ipynb walks through the paper's examples on the published results.

Plot styles

The same profiles (Figure 5 of the paper) in the three ways the package can draw them.

Default matplotlib style
Default
plot_quality_profiles(qp)
Interactive Plotly figure
Interactive (Plotly)
plot_quality_profiles_interactive(qp)
SciencePlots science style
SciencePlots, one journal column
plot_quality_profiles(qp, style="science")
SciencePlots science and ieee styles
SciencePlots + IEEE, works in black and white
style=["science", "ieee"]

The matplotlib figures can be saved as PDF, SVG or PNG; for a paper, prefer PDF. The Plotly figure is saved as an interactive HTML page with fig.write_html(...).

Interactive figures

Hovering shows the value of every solver at the precision level under the cursor:

Moving the cursor along the quality profiles shows the value of every solver

With many solvers, click a name in the legend to hide it, and drag a box to zoom; a double-click goes back to the full view. Here, on the seven derivative-free solvers of Figure 12:

Hiding solvers from the legend and zooming on the high-accuracy range

The gallery is drawn by examples/utils/make_readme_gallery.py.

What is computed

Solvers that start from the same point can end in different local minima. Counting iterations or function evaluations does not show which one found the better solution; quality profiles do.

Two solvers start from the same point: Solver A is fast but stops in a shallow valley, Solver B is slower but finds the deeper one; the difference between the two final values is the quality gap

For every solver s and precision level τ ∈ [0, 1], the quality profile Qs(τ) is the fraction of test problems on which s reduced the gap between the starting value f(x0) and a reference value fL (by default the best value found by any solver) by at least a factor τ. The package computes the scaled profiles [Qs(τr1)]r2: r1 zooms the abscissa, r2 the ordinate, and neither changes the ranking of the solvers.

All the package needs is one final value per problem and solver, plus the value at the common starting point:

Three steps: run the solvers on the same problems from the same starting points, collect one final value per problem and solver, read the quality profiles, where a higher curve means better solutions

On a semilog plot, the value at precision 10−k is the fraction of problems on which a solver gained at least k orders of magnitude; the higher the curve, the better.

The definitions, weak and strong profiles, the choice of reference value, the scaling, the properties proved in the paper and the handling of failures are described in docs/what-is-computed.md.

API

Every parameter, return value and error is documented in docs/api.md.

The paper's experiments

examples/data contains the results published with the paper, one CSV file per experiment: the CUTEst problem and its dimension n, the starting value f0, then one column per solver. examples/utils/paper_data.py loads them, also restricted to one problem dimension, e.g. load("cutest_smooth", n=10000).

Datasets
File Problems Solvers Paper figures
cutest_smooth.csv 166 L-BFGS, CG+, TRON, CG_DESCENT 4–8
negative_curvature.csv 166 TN-NONegCurv, TN-NC1, TN-NC2, TN-NC3 9–11
dfo_budget_1000n.csv 145 NEWUOA and six Direct Search methods 12
dfo_budget_750n.csv 145 same 13
dfo_budget_500n.csv 145 same 14
dfo_budget_250n.csv 145 same 15

Where the data come from, one correction made to the MATLAB files, and two known issues in the original data are described in examples/data/README.md.

uv run python examples/reproduce_paper_figures.py    # redraw the paper's figures
uv run python examples/compare_plot_styles.py        # one figure in every plot style
uv run python examples/try_plot_styles.py --help     # SciencePlots and Plotly on every dataset

Development

uv sync                          # environment with Plotly, SciencePlots and the dev tools
uv run pytest                    # unit tests and doctests
uv run pytest --nbmake notebooks # execute the example notebook
uv run ruff check . && uv run ruff format --check .
uv run mypy                      # strict type checking
uv build                         # build the sdist and the wheel into dist/

The tests check every result against a line-by-line port of the MATLAB algorithm, and check the properties proved in the paper.

Authors

Quality Profiles is developed and maintained by Christian Piermarini (Sapienza University of Rome).

Collaborators: Giovanni Fasano (Ca' Foscari University of Venice) and Massimo Roma (Sapienza University of Rome), co-authors of the paper that introduces the method.

Citation

If you use this package, please cite the paper (see CITATION.cff):

@article{FasanoPiermariniRoma2026QualityProfiles,
  title   = {Benchmarking Optimization Algorithms with Quality Profiles and Test Set Profiles},
  author  = {Fasano, Giovanni and Piermarini, Christian and Roma, Massimo},
  journal = {Mathematical Programming Computation},
  note    = {Accepted. arXiv:2607.16798},
  doi     = {10.48550/arXiv.2607.16798},
  year    = {2026}
}

Questions and contributions

Questions and contributions are very welcome. Feel free to open an issue for any question, bug report or idea, or to send a pull request following the guidelines for contributing.

License

MIT. See LICENSE.

Metadata

Release files for quality-profiles 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for quality-profiles 1.0.0
File Size Uploaded Provenance Actions
quality_profiles-1.0.0.tar.gz 1.6 MB With provenance Details for quality_profiles-1.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for quality-profiles 1.0.0
File Interpreter ABI Platform Actions
quality_profiles-1.0.0-py3-none-any.whl Python 3 none any Details for quality_profiles-1.0.0-py3-none-any.whl

Total release size: 1.6 MB

Release files / quality_profiles-1.0.0.tar.gz

Download URL quality_profiles-1.0.0.tar.gz
Size 1.6 MB
Tags Source
SHA-256 checksum
How to use checksums
56edd4e080441f5a5c22cf5781fb26cc38dce0ff9c57ebfca32d68abd0252d98
BLAKE2b-256 checksum
How to use checksums
1f5dd24406fd16ee34f750703829cdfa8027de6b3a70b8851799aa2ab22012eb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Release files / quality_profiles-1.0.0-py3-none-any.whl

Download URL quality_profiles-1.0.0-py3-none-any.whl
Size 24.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8ba49c58d60aa3bb12ec8e9be0105b9c90ebe744c75cf2dc7b932fc025ccf215
BLAKE2b-256 checksum
How to use checksums
12bd7b389400a40fc84c8d24bc6659acabf6d01272514772906daf6fe7f2ab4e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Security for quality-profiles 1.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for quality_profiles-1.0.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page