numoptlib 0.1.1

An optimization library with gradient descent and Newton methods

numoptlib

A Python library implementing classical numerical optimization algorithms from scratch. This project was developed alongside graduate coursework in computational geophysics with the goal of providing clean, well-tested implementations of gradient-based optimization methods, convergence diagnostics, and benchmarking tools.

Motivation

Most scientific Python code treats optimization as a black box: you pass a function to scipy.optimize.minimize and receive a solution. This library is an attempt to look inside that box by implementing optimization algorithms from first principles and exploring the tradeoffs between different methods.

The implementations are based primarily on:

Nocedal, J. & Wright, S. (2006). Numerical Optimization (2nd ed.)

Implemented Methods

Method	Type	Line Search	Convergence
Gradient Descent	Unconstrained	Strong Wolfe	O(1/k)
Momentum	Unconstrained	Strong Wolfe	O(1/k)
Adam	Unconstrained	None	Adaptive
Newton	Unconstrained	Strong Wolfe	Quadratic
BFGS	Unconstrained	Strong Wolfe	Superlinear
Projected Gradient Descent	Constrained	Strong Wolfe	O(1/k)
Augmented Lagrangian	Constrained	BFGS Subproblem	Linear

Installation

Install directly from PyPI:

pip install numoptlib

Or install the development version:

git clone https://github.com/Kripa-Vyas03/numoptlib
cd numoptlib
pip install -e .

Example Usage

The example below minimizes the Rosenbrock function using BFGS.

import numpy as np
from numoptlib.unconstrained.bfgs import bfgs

def rosenbrock(x):
    return 100 * (x[1] - x[0]**2)**2 + (1 - x[0])**2

def rosenbrock_grad(x):
    return np.array([
        -400 * x[0] * (x[1] - x[0]**2) - 2 * (1 - x[0]),
        200 * (x[1] - x[0]**2)
    ])

x0 = np.array([-1.1, 1.1])

result = bfgs(
    rosenbrock,
    rosenbrock_grad,
    x0,
    max_iter=2000
)

print(result.x)
print(result.fun)

Typical output:

Solution: [0.99999998 0.99999996]
Function value: 0.000000
Converged: True
Iterations: 34

Additional examples, including visualization and benchmarking scripts, can be found in the examples/ directory.

Benchmarks

The algorithms were benchmarked on the Rosenbrock function and several quadratic optimization problems.

Rosenbrock Function

Start Point	Gradient Descent	Newton	BFGS	Momentum	Adam
[-1, 1]	847	20	1	1	1034
[0, 0]	>2000	13	21	198	>2000
[-1.1, 1.1]	>2000	21	16	339	1195

Additional benchmark results are available in the repository documentation.

Running Tests

Run the test suite with:

python -m pytest tests/ -v

The tests include:

1. Correctness tests
2. Result object validation
3. Method-specific convergence tests

Dependencies

• Python >= 3.9
• NumPy
• pytest (testing)
• Matplotlib (examples and benchmarking)

References

Nocedal, J., & Wright, S. (2006). Numerical Optimization. Springer.

Boyd, S., & Vandenberghe, L. (2004). Convex Optimization. Cambridge University Press.