qqn-torch

QQN (Quadratic Quasi-Newton) — a drop-in replacement for torch.optim.LBFGS that searches a quadratic path blending the steepest-descent and quasi-Newton directions.

What is QQN?

Classic quasi-Newton methods (like L-BFGS) take a single direction -H∇f and line-search along it. This works well near a minimum but can be fragile far from it, where the curvature approximation is unreliable.

QQN instead constructs a quadratic path that interpolates between the steepest-descent direction and the quasi-Newton direction:

d(t) = t(1-t)·(-∇f) + t²·(-H∇f)

with t ∈ [0, 1]. The key properties are:

Because the path starts tangent to -∇f, the beginning of the path always decreases f (when ∇f ≠ 0). This anchors global convergence, while the t = 1 endpoint recovers L-BFGS superlinear behavior near the optimum. The line search walks t directly and discovers the right blend — no manual tuning of a mixing coefficient.

Installation

pip install qqn-torch

Or from source:

git clone https://github.com/your-org/qqn-torch
cd qqn-torch
pip install -e .

Requires PyTorch.

Quick Start

QQN follows the same closure-based API as torch.optim.LBFGS:

import torch
from qqn_torch import QQN

# A simple quadratic objective.
x = torch.tensor([1.5, -2.0], requires_grad=True)

optimizer = QQN([x], max_iter=20)


def closure():
    optimizer.zero_grad()
    loss = (x[0] - 3.0) ** 2 + (x[1] + 1.0) ** 2
    loss.backward()
    return loss


for _ in range(10):
    loss = optimizer.step(closure)
    print(f"loss = {loss:.6e}, x = {x.detach().tolist()}")

The closure must:

1. Clear gradients (zero_grad()),
2. Compute the loss,
3. Call loss.backward(),
4. Return the loss.

This is identical to the torch.optim.LBFGS contract, so existing LBFGS training loops work unchanged.

Configuration

QQN(
    params,
    history_size=10,  # L-BFGS curvature-pair history
    line_search="armijo",  # "armijo" | "backtracking" | "strong_wolfe" | "fixed"
    oracle="lbfgs",  # "lbfgs" | "momentum" | "secant" | Oracle instance
    region=None,  # None | "box" | "trust" | Region instance
    max_iter=20,  # inner iterations per .step()
    tol_grad=1e-7,  # gradient-norm stopping tolerance
    tol_change=1e-9,  # step/objective change tolerance
    line_search_options=None  # dict forwarded to the line search (c1, c2, ...)
)

Four orthogonal, swappable components

QQN is built as a combiner of four independent pieces. Each can be changed without touching the rest.

1. Oracle — the t = 1 endpoint (-H∇f)

Because the steepest-descent contribution anchors convergence, the oracle is free to be aggressive — it need not guarantee descent on its own.

2. Line search — walks the path and picks t

The line search is not an implementation detail — it is the glue that makes the gradient and oracle work together. Convergence quality is bounded by line-search quality.

3. Region — optional projection of candidate points

When a region is active, the line search navigates the projected path d_R(t) = project_R(x, x + d(t)) - x, so descent guarantees hold on the feasible set. Custom regions can be composed with SequentialRegion.

4. Gradient

The raw -∇f signal, the path's tangent at the origin.

Custom components

You can pass instances instead of string shortcuts for full control:

from qqn_torch import QQN
from qqn_torch.regions import TrustRegion
from qqn_torch.oracles import SecantOracle

optimizer = QQN(
    params,
    oracle=SecantOracle(alpha_init=0.5),
    region=TrustRegion(radius=2.0, max_radius=1e3),
    line_search="strong_wolfe",
    line_search_options={"c1": 1e-4, "c2": 0.9},
)

How it works (per inner iteration)

g       = flat_grad(closure)         # autograd on the closure
qn_dir  = oracle.direction(g, state) # the t=1 endpoint, -H g
grad_dir = -g                        # the path tangent at t=0
d(t)    = t(1-t)·grad_dir + t²·qn_dir # quadratic path
t*      = line_search(...)           # picks the blend AND the step
x      += project(d(t*))             # apply (optionally projected) step
oracle.update(s, y)                  # s = Δx, y = Δg
region.update(...)                   # e.g. adapt trust radius

Advantages

• Adaptive: automatically balances conservative vs. aggressive steps.
• Robust: d'(0) = -∇f plus line-search fallbacks ensure progress even when the oracle is poor.
• Efficient: L-BFGS acceleration when curvature is reliable.
• Modular: gradient, oracle, search, and region are independently swappable.

Limitations

• Memory: stores L-BFGS history (O(m·n)).
• Overhead: walking the curved path adds modest per-iteration cost.
• Tuning: sensitive to history size, line-search constants, region radii.
• Line-search sensitivity: a poor line search undermines convergence and the quality of the curvature updates.

Theoretical guarantees

Under standard assumptions (smooth objective, bounded gradients):

• Global convergence — anchored by the steepest-descent tangent.
• Superlinear convergence — inherited from L-BFGS when t → 1 near the optimum.
• Descent property — every accepted step decreases f, enforced by the line search's sufficient-decrease test.

All guarantees are contingent on the line search satisfying sufficient decrease. When a region is active, they hold on the projected path d_R(t).

Documentation

• algorithm.md — comprehensive algorithm reference.

Results

To reproduce, execute run_reports.js.

Sigmoid

The sigmoid σ(x) = 1/(1+e⁻ˣ) squashes any real input into (0, 1). It is smooth and historically popular, but saturates for large |x|, where its gradient vanishes — a classic source of slow learning in deep nets.

Results: QQN reaches the lowest final loss (1.75e-01, 98.9% train / 87.1% test) and is the only optimizer to break the 0.2 milestone (337 iters). Adam trails at 2.47e-01 but attains the best test accuracy (87.6%), hinting that QQN's lower training loss edges toward mild overfitting. L-BFGS stalls badly at 1.12e+00 — the sigmoid's saturation flattens the curvature signal, so its quasi-Newton step never gains traction beyond the first milestone.

Sine

Sine sin(x) is a bounded, periodic, infinitely differentiable activation. Its oscillatory nature can encode periodic structure, but the non-monotonicity makes optimization landscapes bumpy and initialization sensitive.

Results: Sine is a standout case: both QQN (4.41e-02) and L-BFGS (4.60e-02) drive the loss an order of magnitude below Adam (3.48e-01) and reach 100% train accuracy. QQN just edges out L-BFGS on final loss while tracking it closely at every milestone, and both share the best AUC scores (-0.92 / -0.96) in the whole suite. The smooth, well-conditioned curvature of the sine landscape lets the quasi-Newton machinery shine — Adam converges fast early but plateaus.

ReLu

The ReLU max(0, x) is the modern default: cheap, non-saturating for positive inputs, and inducing sparse activations. Its downside is the "dying ReLU" problem — units stuck at zero receive no gradient.

Results: QQN wins decisively (1.15e-01, 98.9% train) and is the only method to pass the 0.2 milestone (263 iters). Adam sits at 2.34e-01, while L-BFGS again stalls (8.98e-01, 66.7% train) — the ReLU kink and dying-unit zero-gradient regions corrupt its curvature estimate. This is the archetypal QQN advantage: the steepest-descent tangent keeps it moving where pure L-BFGS gets stuck, yet it still exploits curvature once past the rough zone.

Abs

The absolute value |x| is a symmetric, V-shaped activation. Like ReLU it is piecewise linear and non-saturating, but it folds the input rather than clipping it, preserving magnitude information from both signs.

Results: QQN leads (1.82e-01, 94.9% train, 86.9% test), reaching the 0.2 milestone at 320 iters where L-BFGS (6.07e-01) and Adam (9.81e-01) fall short. Notably Adam's trajectory shows instability — it dips to -0.68 (log10) then spikes back to +1.84 before recovering, a sign the piecewise-linear fold produces a jagged landscape that trips the adaptive step sizes. QQN's line-searched blend absorbs these irregularities gracefully.

LogAbs

The symmetric log sign(x)·ln(1+|x|) is an odd, monotonic activation that grows logarithmically in both directions. Like abs it is symmetric and preserves sign, but its compressive log growth tames large magnitudes without fully saturating, keeping a nonzero gradient everywhere.

Results: The compressive log growth restores a usable curvature signal: L-BFGS recovers to first place (1.64e-01) with QQN a close second (1.94e-01), and both reach the 0.2 milestone (176 / 214 iters) while Adam (3.51e-01) does not. Compared to plain abs, keeping a nonzero gradient everywhere clearly benefits the quasi-Newton methods — L-BFGS goes from stalling on abs to leading on logabs.

Gaussian

The Gaussian e⁻ˣ² is a bump centered at the origin, peaking at 1 and decaying to 0 in both directions. It yields localized, radial-basis-like responses but saturates quickly away from zero.

Results: L-BFGS narrowly wins (9.54e-02) over QQN (1.23e-01), and it is the only optimizer to clear the 0.1 milestone (133 iters). This activation is the most expensive per iteration (~450 ms/it), so all methods complete only ~130–160 iterations in the budget. The smooth, radially localized responses give clean curvature that L-BFGS exploits fully; QQN matches its milestone pace but its extra path-walking overhead costs a slight edge here.

Tanh

Tanh tanh(x) is a zero-centered sigmoid mapping to (-1, 1). The zero-centering often speeds convergence relative to the logistic sigmoid, but it shares the same saturation-induced vanishing-gradient issue.

Results: Another strong quasi-Newton case: QQN (5.41e-02) and L-BFGS (6.30e-02) both hit 100% train accuracy and reach the 0.1 milestone (226 / 200 iters), while Adam lags at 2.89e-01. The zero-centered, smooth tanh landscape is well-conditioned, so — as with sine — the curvature-driven methods dominate. QQN edges L-BFGS on final loss, mirroring the sine result.

Swish

Swish x·σ(x) is a smooth, non-monotonic self-gated activation. It behaves like ReLU for large positive x but dips slightly below zero for small negatives, which often improves deep-network training.

Results: Here Adam takes the lead (1.87e-01) and is the sole optimizer to pass the 0.2 milestone (328 iters); QQN follows at 2.48e-01 but never clears 0.5→0.2. L-BFGS collapses entirely (2.31e+00, 15% train) — the smooth but non-monotonic self-gating apparently yields curvature that misleads the pure quasi-Newton step. QQN stays robust thanks to its gradient anchor but its oracle inherits the same poor curvature, blunting its usual advantage.

GeLu

GELU x·Φ(x) (Φ is the standard-normal CDF) weights inputs by their probability of being positive. It is the smooth, probabilistic cousin of ReLU and the default in modern transformer architectures.

Results: QQN wins the Pareto frontier outright (2.14e-01) but neither it nor Adam (2.67e-01) reaches the 0.2 milestone; Adam does clear it internally (283 iters) while QQN does not. L-BFGS diverges catastrophically (2.31e+00, 13.6% train — essentially random). As with Swish, the smooth-but-non-monotonic GELU produces curvature that sabotages raw L-BFGS, and QQN, sharing the same L-BFGS oracle, only partially escapes.

Elu

The ELU x if x>0 else α(eˣ−1) (here α=1) is linear for positive inputs and saturates smoothly to −α for negatives. The negative saturation pushes mean activations toward zero, aiding convergence, while avoiding dead units.

Results: QQN dominates (1.07e-01, best AUC -0.62), reaching the 0.2 milestone in just 221 iters versus Adam's 649; Adam finishes at 1.72e-01. L-BFGS stalls at 7.03e-01 (76.5% train). ELU's negative saturation keeps mean activations near zero, and QQN converts that into fast, smooth descent — its loss trajectory drops steadily to -0.97 (log10) with no oscillation.

SeLu

SELU is a scaled ELU (λ·x for positives, λα(eˣ−1) for negatives, with the fixed constants λ≈1.0507, α≈1.6733). These constants make activations self-normalizing, driving outputs toward zero mean and unit variance.

Results: The self-normalizing constants pay off for curvature methods: L-BFGS (7.61e-02) and QQN (9.46e-02) both reach the 0.1 milestone (272 / 357 iters) and 99%+ train accuracy, with the two best AUCs in the ELU family (-0.74 / -0.66). Adam trails far behind (6.59e-01, 80.9% train). Keeping activations at zero mean / unit variance produces exactly the well-scaled curvature that L-BFGS and QQN thrive on.

Leaky ReLu

Leaky ReLU x if x>0 else 0.01x fixes the dying-ReLU problem by allowing a small, nonzero gradient for negative inputs. It keeps every unit learning while retaining ReLU's cheap piecewise-linear form.

Results: QQN leads (1.61e-01) and reaches the 0.2 milestone (308 iters), closely followed by Adam (2.40e-01, 278 iters). L-BFGS fails completely (2.33e+00, 9.7% train — worse than chance), stuck at its initial loss the entire run. The small negative slope that cures dying-ReLU units does not rescue raw L-BFGS's curvature estimate, but QQN's steepest-descent anchor once again keeps it converging where pure quasi-Newton cannot.

SoftPlus

Softplus ln(1+eˣ) is a smooth approximation to ReLU. It is everywhere differentiable and strictly positive, with derivative equal to the sigmoid — but its smoothness costs more compute than plain ReLU.

Results: Unusually, Adam wins here (2.22e-01) and is the only method to reach the 0.2 milestone (749 iters); QQN finishes at 2.74e-01 and L-BFGS stalls (1.48e+00, 39.2% train). The smooth ReLU approximation seems to favor Adam's per-parameter adaptation, while QQN — though robust — makes slower progress and its L-BFGS oracle is again hampered by unreliable curvature.

Mish

Mish x·tanh(softplus(x)) is a smooth, non-monotonic self-gated activation. It behaves like ReLU for large positive x but allows a small negative response for slightly negative inputs, preserving a bit of gradient flow. Its smoothness and self-regularizing shape often improve deep-network training. It is similar to Swish but uses a tanh(softplus(·)) gate instead of a sigmoid.

Results: QQN takes the frontier (1.57e-01, best AUC -0.52) and reaches the 0.2 milestone slightly faster than Adam (283 vs 324 iters); Adam finishes at 2.61e-01 with the top test accuracy (87.5%). L-BFGS collapses to chance (2.31e+00, 14.5% train), consistent with the other smooth self-gated activations (Swish, GELU) where its curvature estimate is unreliable.

Identity

Identity f(x) = x is the simplest activation, passing inputs through unchanged. It is linear and does not introduce nonlinearity, so it is rarely used in hidden layers but can be useful for output layers in regression tasks.

Results: With no nonlinearity the network collapses to a linear model, so accuracies cap around 82%. QQN (2.92e-01) and L-BFGS (3.13e-01) perform near-identically — expected, since the objective is essentially convex and their curvature paths coincide. Adam diverges wildly (9.39e+00, 37.6% train), its loss exploding to +0.97 (log10) late in training — a cautionary reminder that adaptive methods can destabilize even on the simplest landscape.

Triangle

The triangle wave is a periodic, piecewise-linear activation that ramps linearly up and down between -1 and 1. Unlike smooth periodic activations such as sine, its constant-magnitude slopes give uniform gradients away from the turning points, but the sharp corners introduce non-differentiable kinks.

Results: All three optimizers do well here — L-BFGS (6.61e-02), QQN (7.69e-02), and Adam (1.00e-01) all reach 100% (or near-100%) train accuracy and the best test accuracies of the suite (~87.7%). L-BFGS and QQN clear the 0.1 milestone (140 / 148 iters). Despite the non-differentiable corners, the constant-magnitude slopes give uniform, informative gradients that all methods exploit; the per-iteration cost is high (~340 ms/it) so runs stay short.

Sawtooth

The sawtooth wave is a periodic ramp that rises linearly from -1 to 1 and then drops discontinuously back down. It shares the triangle's constant slope but adds a hard discontinuity each period, making it the most optimization-hostile of the periodic activations while still encoding periodic structure.

Results: As predicted, the sawtooth is the most hostile activation: no optimizer reaches even the 1.0 milestone. QQN does best (1.97e+00, 31.3% train) but manages only 56 iterations — its line search burns evaluations fighting the periodic discontinuities (~1075 ms/it). L-BFGS (2.14e+00) and Adam (3.01e+00, ~10% train, i.e. chance) fare worse. The hard per-period jump destroys gradient continuity, and while QQN degrades most gracefully, the landscape defeats all first- and second-order methods alike.

Results Summary

Across the 18 activation functions benchmarked on Fashion-MNIST MLPs, clear patterns emerge that validate QQN's design philosophy.

Win counts (best final loss)

*Starred entries are near-ties where the two curvature methods finish within noise of each other.

Key takeaways

1. QQN is the most consistent winner or runner-up. It never collapses to chance and reaches the target milestone on nearly every activation. Even when it doesn't win outright (Swish, Softplus, Gaussian), it stays within striking distance and remains stable.
2. L-BFGS is high-variance. On well-conditioned, smooth landscapes (sine, tanh, logabs, Gaussian, SELU, triangle) it is competitive or best. But on ReLU-family kinks and smooth-but-non-monotonic gates (Swish, GELU, Mish, leaky-ReLU) its curvature estimate is corrupted and it diverges to chance-level accuracy (~10–15% train). This is the failure mode QQN was built to fix.
3. QQN's gradient anchor is decisive. In every case where L-BFGS stalls or collapses (ReLU, abs, leaky-ReLU, Swish, GELU, Mish, ELU, SELU-poor regions), QQN keeps converging because the d'(0) = -∇f tangent guarantees progress regardless of oracle quality — while still exploiting curvature once past the rough zone.
4. Adam wins only the smooth-ReLU approximations (Swish, Softplus), where per-parameter adaptation suits the landscape, but it can destabilize badly (Identity: diverges to 9.39; abs: oscillates; sawtooth: chance).
5. The landscape can defeat everyone. On the discontinuous sawtooth, no method clears even the 1.0 milestone — though QQN still degrades most gracefully.

Conclusion

The empirical results confirm QQN's central thesis: blending the steepest-descent tangent with a quasi-Newton endpoint yields the best of both worlds. QQN inherits L-BFGS's superlinear speed on well-conditioned, curvature-rich problems (sine, tanh, SELU) while its gradient anchor prevents the catastrophic failures that sink raw L-BFGS on kinked or non-monotonic activations (ReLU, leaky-ReLU, Swish, GELU, Mish).

In practical terms:

• Choose QQN as a robust default. It matches or beats L-BFGS almost everywhere and never diverges, making it a safe drop-in replacement.
• Prefer L-BFGS only when the landscape is known to be smooth and well-conditioned, where it may shave a small margin off the final loss.
• Reach for Adam for smooth ReLU-like activations (Swish, Softplus), but watch for its instability on simple or non-smooth objectives.

The modular design — swappable oracle, line search, and region — means these trade-offs can be tuned per problem without rewriting the optimizer. When in doubt, QQN's line-searched quadratic path adapts automatically, discovering the right blend between caution and aggression on each iteration.

Acknowledgements

The strong-Wolfe line search is adapted from PyTorch's _strong_wolfe helper (torch/optim/lbfgs.py, BSD-licensed). See qqn_torch/_vendor/strong_wolfe.py for attribution.

License

See the LICENSE file. Vendored code retains its original PyTorch BSD license.