polybench

Multivariate polynomial arithmetic benchmark tests.

Many scientific and engineering applications utilise multivariate polynomial arithmetic in their algorithms and solutions. Here we provide a set of benchmark tests for often-used operations in multivariate polynomial arithmetic:

• Greatest common divisor
• Factorisation

Requirements

• Python >= 3.6.1

You also need at least one tool to be benchmarked. They are (in alphabetical order):

• Fermat: uses fer64 on PATH by default; set FERMAT_COMMAND to specify a different executable.
• FLINT: automatically downloaded and built via vcpkg (requires CMake >= 3.15, a C compiler and the Make utility; see also vcpkg dependencies).
• FORM: uses form on PATH by default, downloading a release binary if not found; set FORM_COMMAND to specify a different executable.
• Mathematica: Free Wolfram Engine for Developers is sufficient to run. Uses wolframscript on PATH by default; set WOLFRAMSCRIPT_COMMAND to specify a different WolframScript executable.
• Poly Mojo: --polymojo measures ordinary integer GCD and factorisation; --polymojocertified includes certificate construction in the measured operation and verifies each certificate, but excludes verification time from the reported measurement. Poly Mojo and its matching Mojo compiler are automatically downloaded via Pixi (required on PATH); the included configuration targets Linux x86-64. To use existing precompiled packages instead of this setup, set POLY_MOJO_PATH to an absolute directory containing poly.mojoc, polynomials.mojoc, and apn_mojo.mojoc. In this case the matching compiler defaults to mojo on PATH and can be selected with MOJO_COMMAND. In either case, a polybench benchmark program is built using the packages.
• reFORM: automatically downloaded and built (requires Rust >= 1.36).
• Rings: automatically downloaded; the benchmark program is built (requires JDK >= 8).
• Singular: uses Singular on PATH by default; set SINGULAR_COMMAND to specify a different executable.
• Symbolica: automatically downloaded and built (requires Rust >= 1.96), running in restricted mode. Built with optimization level 3, full link-time optimization, one codegen unit, and target-cpu=native, with Symbolica's default faster allocator and GMP backend. The executable is optimized for the build machine; rebuild it on each benchmark machine. Set RUSTFLAGS to override the CPU settings when needed.

Getting started

Clone this repository and try to run the run.sh script:

git clone https://github.com/tueda/polybench.git
cd polybench
./run.sh --all

When starting the script for the first time, it automatically sets up a virtual environment for required Python packages so that it will not dirty your environment. Some of the tools are provided as libraries registered in public package registries, so the first run takes some time to download, compile and link them with test binaries. After testing, a CSV file and comparison plots will be generated.

For practical benchmarking, configuration parameters should be set adequately. See the help message shown by

./run.sh --help

You can also use pip, pipx, Poetry or Docker with this repository. Installation with pip(x) install or poetry install makes a command polybench available, which acts as the run.sh script described above.

pip install polybench
polybench --all
python -m polybench --all  # alternative way to launch

pipx install polybench
polybench --all

git clone https://github.com/tueda/polybench.git
cd polybench
poetry install
poetry run polybench --all

Docker

The Dockerfile supports only linux/amd64/v3.

Build the image:

docker build --platform linux/amd64/v3 -t polybench .

Containers start in /work. Run with results kept inside the disposable container:

docker run --rm polybench --all

To save results in the current directory on the host, mount it at /work and use your UID/GID:

docker run --rm --user "$(id -u):$(id -g)" -v "$PWD:/work" polybench --all

To start a shell with the same environment, use either:

docker run --rm -it polybench bash

docker run --rm -it --user "$(id -u):$(id -g)" -v "$PWD:/work" polybench bash

The image includes prepared solvers and Poly Mojo's 5- and 8-variable drivers; other variable counts compile offline when first used in a container.

Fermat and Mathematica/Wolfram Engine are not included.

Fermat

To use a host installation of Fermat, mount its directory and set FERMAT_COMMAND. For example:

docker run --rm \
  -v /path/to/fermat-installation:/opt/fermat:ro \
  -e FERMAT_COMMAND=/opt/fermat/fer64 \
  polybench --fermat

Wolfram Engine

To use a host installation of Wolfram Engine, mount WolframScript and the Engine, and retain a separate directory for the container's license. Use your host UID/GID for both activation and subsequent runs so that the license files remain accessible. For example, with Wolfram Engine 15.0:

mkdir -p wolfram-license

docker run --rm -it --user "$(id -u):$(id -g)" \
  -v /opt/Wolfram/WolframScript:/opt/Wolfram/WolframScript:ro \
  -v /usr/local/Wolfram/WolframEngine/15.0:/usr/local/Wolfram/WolframEngine/15.0:ro \
  -v "$PWD/wolfram-license:/home/polybench/.WolframEngine/Licensing" \
  -e WOLFRAMSCRIPT_KERNELPATH=/usr/local/Wolfram/WolframEngine/15.0/Executables/WolframKernel \
  polybench /opt/Wolfram/WolframScript/bin/wolframscript -activate

docker run --rm --user "$(id -u):$(id -g)" \
  -v /opt/Wolfram/WolframScript:/opt/Wolfram/WolframScript:ro \
  -v /usr/local/Wolfram/WolframEngine/15.0:/usr/local/Wolfram/WolframEngine/15.0:ro \
  -v "$PWD/wolfram-license:/home/polybench/.WolframEngine/Licensing" \
  -e WOLFRAMSCRIPT_KERNELPATH=/usr/local/Wolfram/WolframEngine/15.0/Executables/WolframKernel \
  -e WOLFRAMSCRIPT_COMMAND=/opt/Wolfram/WolframScript/bin/wolframscript \
  polybench --mathematica

See Wolfram's official Docker instructions for activation and licensing details.

Example

Additional benchmark results are available here.

Development

# Initialisation
poetry install
poetry run pre-commit install

# Linting and testing
poetry run pre-commit run --all-files
poetry run pytest

# Linting and testing for Cargo subproject
cd path/to/project
cargo fmt
cargo clippy
cargo test

# Linting and testing for Gradle subproject
cd path/to/project
./gradlew spotlessApply
./gradlew check

# Test run
./run.sh <options>  # for example, --all

# Release a new version
./scripts/make-release.sh <new_version>  # for example, 0.3.0rc1

License

MIT