ZoSolvers

Zeroth-order optimisation solvers with Gaussian and sphere random oracles.

ZoSolvers provides gradient-free solvers for minimisation and minimax problems. When the gradient of the objective is unavailable — because the function is non-differentiable, comes from a black-box simulator, or is too expensive to differentiate — ZoSolvers estimates it using random directional perturbations.

📖 Full documentation: ZoSolvers Manual (PDF)

Install

pip install ZoSolvers

Requirements: Python ≥ 3.10, NumPy ≥ 2.2, Matplotlib ≥ 3.7

---

Features

• Two oracle types — Gaussian (u ~ N(0, B⁻¹)) and sphere (uniform on the unit sphere), both with optional preconditioning via a precision matrix B
• Four solvers — ZOGD, ZOEGm (minimisation), ZOGDA, ZOEGmm (minimax)
• Three finite-difference methods — forward, backward, centered
• Flexible mini-batching — fixed number of oracle samples per step, or a growing schedule (t="iteration")
• Constrained problems — pass any projection function, and choose whether the initial guess is projected too
• Early stopping — based on relative function decrease or oracle norm
• Parallel mini-batches — the t oracle samples per step are evaluated across n_jobs workers (thread or process backend)
• Efficient sampling — Cholesky factorisation cached at construction; diagonal B handled without matrix inversion

---

Quick Start

Minimisation

import numpy as np
from ZoSolvers.minimisation import ZO_gauss_min

def f(x):
    return x[0]**2 + x[1]**2 + x[0]*x[1]

x0  = np.array([5.0, -5.0])
opt = ZO_gauss_min(f, x0, h=1e-2, mu=1e-5, N=2000, t=10)

# Zeroth-order gradient descent
x_traj = opt.ZOGD(method="center")

# Zeroth-order extra-gradient
x_traj = opt.ZOEGm(method="center", gamma=1.0)

Minimax

from ZoSolvers.minimax import ZO_gauss_minmax

def f(x, y):
    return x[0]**2 + x[1]**2 - y[0]**2 - y[1]**2 + x[0]*y[1]

x0 = np.array([5.0, -5.0])
y0 = np.array([3.0, -3.0])

opt = ZO_gauss_minmax(f, x0, y0, h=1e-3, tau=1, mu=1e-8, N=10000, t=5)

# Gradient descent-ascent
x_traj, y_traj = opt.ZOGDA(method="center")

# Extra-gradient
x_traj, y_traj = opt.ZOEGmm(method="center", gamma=0.8)

Sphere Oracle and Precision Matrix

B = np.array([[10.0, 0.5],
              [0.5,  2.0]])

opt = ZO_gauss_min(f, x0, h=1e-2, mu=1e-5, N=2000, t=10,
                   B=B, oracle_type="sphere")
x_traj = opt.ZOGD(method="center")

Parallel Function Evaluations

The t oracle samples that make up one step are independent, so they can be evaluated concurrently. Pass n_jobs (-1 = all cores):

opt = ZO_gauss_min(f, x0, h=1e-2, mu=1e-5, N=2000, t=32, n_jobs=-1)
x_traj = opt.ZOGD(method="center")
opt.close()          # or use the solver as a context manager

The worker pool is created once and reused across every iteration. Use the solver as a context manager to shut it down automatically:

with ZO_gauss_minmax(f, x0, y0, h=1e-3, mu=1e-8, N=10000, t=16, n_jobs=8) as opt:
    x_traj, y_traj = opt.ZOGDA(method="center")

Two backends are available:

backend	Use when	Notes
"thread" (default)	func releases the GIL — NumPy-heavy models, compiled extensions, subprocess or network-backed simulators	Accepts any callable, including lambdas and closures
"process"	func is pure-Python and CPU-bound	func must be picklable (module-level, not a lambda); np.random.seed no longer makes runs reproducible, since each worker seeds itself

Parallelism pays off when the cost function is expensive relative to the scheduling overhead, the usual case for black-box simulators.

Box-Constrained Problem

proj = lambda x: np.clip(x, -3.0, 3.0)

opt = ZO_gauss_min(f, x0, h=1e-2, mu=1e-5, N=2000, t=10, proj=proj)
x_traj = opt.ZOGD(method="center")

Every iterate the solver computes is feasible, but the first row of the returned trajectory is the x0 you passed in, returned untouched. If x0 lies outside the feasible set, that first row is infeasible. Set project_init=True to project it as well, so the whole trajectory is feasible:

opt = ZO_gauss_min(f, x0, h=1e-2, mu=1e-5, N=2000, t=10, proj=proj,
                   project_init=True)
x_traj = opt.ZOGD(method="center")   # x_traj[0] == proj(x0)

The default is False, which keeps x0 visible exactly as given. For minimax, the same flag covers both x0 and y0, each through its own projection.

---

Solvers

Solver	Class	Problem	Description
ZOGD	ZO_gauss_min	Minimisation	Zeroth-order gradient descent
ZOEGm	ZO_gauss_min	Minimisation	Zeroth-order extra-gradient
ZOGDA	ZO_gauss_minmax	Minimax	Zeroth-order gradient descent-ascent
ZOEGmm	ZO_gauss_minmax	Minimax	Zeroth-order extra-gradient minimax

---

Key Parameters

Parameter	Description
h	Step size
mu	Smoothing parameter for finite differences
N	Maximum number of iterations
t	Oracle samples per step (int or "iteration" for growing schedule)
B	Precision matrix shaping the perturbation distribution (None = identity)
oracle_type	"gaussian" or "sphere"
proj	Projection onto the feasible set (None = unconstrained)
project_init	Project the initial guess onto the feasible set too (default False)
n_jobs	Workers used to evaluate the t samples per step (1 = sequential, -1 = all cores)
backend	"thread" (default) or "process"
tau	(Minimax) Step-size ratio: x-step = h/tau, y-step = h
gamma	(Extra-gradient) Second-stride scale factor

---

Package Structure

ZoSolvers/
├── src/ZoSolvers/
│   ├── minimisation.py   # ZO_gauss_min
│   ├── minimax.py        # ZO_gauss_minmax
│   └── utils.py          # shared utilities
├── tests/
│   ├── min_test.py       # pytest suite for minimisation
│   ├── minimax_test.py   # pytest suite for minimax
│   ├── demo_min.py       # minimisation demo with plots
│   └── demo_minimax.py   # minimax demo with plots
├── Docs/
│   └── ZoSolvers_Manual.pdf
└── pyproject.toml

---

License

MIT