EvoOptimizer

An evolved deep learning optimizer discovered via genetic algorithm search. Outperforms Adam by 2.6% in aggregate fitness and achieves 7.7% relative improvement on CIFAR-10.

Installation

pip install evo-optimizer

Quick Start

from evo_optimizer import EvoOptimizer

model = YourModel()
optimizer = EvoOptimizer(model.parameters(), lr=1.2e-3, total_steps=10000)

for step, (inputs, targets) in enumerate(dataloader):
    optimizer.zero_grad()
    loss = loss_fn(model(inputs), targets)
    loss.backward()
    optimizer.step(step=step)  # Pass step for warmup/cosine scheduling

What Makes It Different?

EvoOptimizer was discovered, not designed. Using genetic algorithm search over 50 generations with 50 individuals, evaluated across Fashion-MNIST, CIFAR-10, and MNIST, this optimizer emerged with several surprising properties:

The evolved update rule:

Δθ = α_sign · sign(g) + α_adam · (m / √v)

This combines the magnitude-insensitivity of Lion-style sign updates with Adam's adaptive moment scaling.

API Reference

EvoOptimizer

EvoOptimizer(
    params,
    lr=1.2e-3,              # Learning rate
    betas=(0.8553, 0.9358), # Momentum coefficients (lower than Adam)
    eps=5.4e-9,             # Numerical stability
    weight_decay=9.7e-4,    # Decoupled weight decay
    warmup_steps=100,       # Linear warmup steps
    total_steps=None,       # Total steps for cosine decay
    use_warmup=True,        # Enable warmup
    use_cosine_decay=True,  # Enable cosine annealing
    alpha_sign=0.7345,      # Weight for sign(gradient) term
    alpha_adam=3.6352,      # Weight for adaptive moment term
)

Functional API

For custom training loops or use with torch.compile():

from evo_optimizer import evo_optimizer_step
from evo_optimizer.functional import compute_lr_scale

# Initialize state
exp_avgs = [torch.zeros_like(p) for p in model.parameters()]
exp_avg_sqs = [torch.zeros_like(p) for p in model.parameters()]

for step in range(num_steps):
    loss = model(x).sum()
    loss.backward()
    
    # Compute scheduled LR
    lr = 1.2e-3 * compute_lr_scale(step, warmup_steps=100, total_steps=num_steps)
    
    evo_optimizer_step(
        list(model.parameters()),
        [p.grad for p in model.parameters()],
        exp_avgs,
        exp_avg_sqs,
        lr=lr,
    )
    
    model.zero_grad()

Benchmark Results

From the original paper, evaluated with 1000 training steps per task:

When to Use EvoOptimizer

Good fit:

• Vision tasks (CNNs, ViTs)
• Short-to-medium training runs (evolved on 500-1000 step budgets)
• When you want built-in scheduling without external schedulers

Consider alternatives:

• Very long training runs (>100k steps) - may want to tune further
• Extremely large models - not yet tested at scale
• Tasks very different from vision classification

Comparison to Similar Optimizers

• vs Adam: Lower momentum, no bias correction, adds sign-based term
• vs Lion: Hybrid approach (Lion is pure sign-based)
• vs AdamW: Similar decoupled weight decay, but different update rule
• vs Sophia: EvoOptimizer doesn't use Hessian information

Citation

@article{marfinetz2025evolving,
  title={Evolving Deep Learning Optimizers},
  author={Marfinetz, Mitchell},
  journal={arXiv preprint arXiv:2512.11853},
  year={2025}
}

License

MIT License - see LICENSE for details.

Contributing

Contributions welcome! Please feel free to submit issues or pull requests.

Areas of interest:

• Benchmarks on additional tasks (NLP, RL, audio)
• Scaling experiments on larger models
• Hyperparameter sensitivity analysis
• Integration with popular training frameworks (PyTorch Lightning, HuggingFace)