evolo-moead 0.1.0

Zero-shot evolutionary architecture search for LoRA via MOEA/D and Gradient Projection Score.

EvoLoRA-MOEAD

Zero-shot evolutionary architecture search for Low-Rank Adaptation (LoRA).

Acknowledgment & Secondary Creation: This repository is an independent engineering implementation and secondary creation based on the theoretical framework proposed by Wu, Neri, and Feng (2026). All necessary original theoretical credit belongs to the authors. DOI: 10.1142/S0129065726500255

evolo-moead searches for performant LoRA adapter configurations — rank r, scaling factor alpha, and insertion sites target_modules — before any fine-tuning begins. The search is training-free: it ranks candidate architectures by the Gradient Projection Score (GPS), a singular-value statistic of the calibration gradients, and explores the trade-off surface with the MOEA/D multi-objective evolutionary algorithm. The recommended configuration is the knee point of the discovered Pareto front and is returned as a ready-to-use peft.LoraConfig.

---

Why

Choosing LoRA hyperparameters by trial-and-error is expensive: every candidate normally requires a full fine-tuning run. EvoLoRA-MOEAD replaces that loop with a single forward/backward pass per candidate and a principled, multi-objective ranking that balances three competing goals:

Objective	Direction	Meaning
MeanGPS	maximise	Average rank-r gradient energy across LoRA layers — a proxy for adapter capacity.
StdGPS	minimise	Inter-layer dispersion of that energy — penalises unstable, lopsided adapters.
TrainableParams	minimise	Adapter parameter budget — honours the parameter-efficiency constraint.

Internally the optimiser minimises the vector [-MeanGPS, StdGPS, TP].

---

Installation

# From PyPI (core search + HuggingFace evaluator)
pip install evolo-moead

# From source, editable, with development and example extras
git clone https://github.com/your-org/EvoLoRA-MOEAD.git
cd EvoLoRA-MOEAD
pip install -e ".[dev,examples]"

Core dependencies: torch>=2.0.0, transformers, peft, pymoo, numpy. Python 3.9+.

---

Quick start

from evolo_moead import EvoLoRATuner

# `model` is any transformers.PreTrainedModel;
# `calibration_loader` yields dict batches containing `labels`.
tuner = EvoLoRATuner(model, calibration_loader)

best_config = tuner.search_best_config(n_gen=50)   # -> peft.LoraConfig

# Feed the result straight into PEFT for the real fine-tuning run.
from peft import get_peft_model
peft_model = get_peft_model(model, best_config)

That is the entire surface. Every evolutionary and linear-algebra detail is hidden behind search_best_config.

Running the bundled example

# CPU-only, no model download — verifies the installation in seconds
python examples/quick_start.py --backend mock

# Real model path (requires the `examples` extra and network access)
python examples/quick_start.py --backend hf --n-gen 20

---

API reference

EvoLoRATuner(model, calibration_dataloader=None, search_space=None, num_calibration_batches=2, task_type=None, evaluator=None)

Argument	Default	Description
model	—	Base model to search adapters for.
calibration_dataloader	None	Loader of labelled dict batches. Required unless evaluator is given.
search_space	None	Override of the default discrete space (see below).
num_calibration_batches	2	Batches drawn per candidate evaluation.
task_type	None	peft task type (e.g. "SEQ_CLS", "CAUSAL_LM").
evaluator	None	Pre-built evaluator; supply a MockEvaluator for CPU-only demos.

search_best_config(n_gen=50, n_partitions=12, seed=42, **moead_kwargs) -> peft.LoraConfig

Runs the full search and returns the knee-point configuration. n_partitions controls the Das-Dennis reference-direction density (for three objectives, n_dirs = (p+1)(p+2)/2, so p=12 yields 91 directions). Extra keyword arguments — n_neighbors, prob_neighbor_mating — are forwarded to the MOEA/D driver.

Default search space

{
    "ranks": [4, 8, 16, 32, 64],
    "alphas": [8, 16, 32, 64],
    "target_modules": [["q", "v"], ["q", "k", "v", "o"], "all-linear"],
}

Override it to match the module-naming convention of the host model. For BERT-style models, for instance, the attention projections are named query / key / value:

search_space = {
    "ranks": [2, 4, 8],
    "alphas": [8, 16],
    "target_modules": [["query", "value"], ["query", "key", "value"]],
}
tuner = EvoLoRATuner(model, loader, search_space=search_space, task_type="SEQ_CLS")

---

How it works

EvoLoRATuner.search_best_config
        │
        ▼
LoRASearchProblem  ──uses──►  BaseEvaluator
   (pymoo, 3-obj)                 ├── HuggingFaceEvaluator  (real gradients)
        │                         └── MockEvaluator         (closed-form surrogate)
        ▼
   run_moead  ──►  MOEADResult  ──►  deduplicate_front
        │                                   │
        ▼                                   ▼
  Das-Dennis ref. dirs              find_knee_point (cosine curvature)
                                            │
                                            ▼
                                      peft.LoraConfig

Gradient Projection Score

For a layer gradient G ∈ ℝ^{d_out × d_in} with singular values σ₁ ≥ σ₂ ≥ …, the rank-r GPS is the Frobenius norm of the optimal rank-r projection:

GPS(G, r) = sqrt( Σ_{i=1}^{r} σ_i² )

Tensors of rank > 2 (convolution kernels, embeddings) are unfolded to 2D with the leading axis as the output dimension. Zero, NaN, and Inf inputs are sanitised, and a CPU float64 fallback guards against non-convergent GPU SVD kernels, so a single degenerate layer never aborts the search.

Knee-point selection

The Pareto front is Min-Max normalised per objective (to prevent the high-magnitude parameter axis from dominating) and sorted along the parameter-count axis. The point whose neighbouring difference vectors subtend the largest angle is returned as the recommended trade-off. Degenerate fronts (fewer than three distinct points) fall back to the minimum-parameter solution.

Memory discipline (HuggingFaceEvaluator)

Each candidate evaluation mounts a temporary adapter with peft.get_peft_model, runs a short forward/backward pass, aggregates per-layer GPS, then unconditionally cleans up in a finally block:

1. zero_grad(set_to_none=True) on the wrapped model,
2. peft_model.unload() to revert every module replacement,
3. gc.collect(), then torch.cuda.empty_cache() and torch.cuda.ipc_collect() when CUDA is present.

This guarantees the host model returns to its original parameter and memory state after every call, so the search loop runs at constant memory.

---

Project layout

EvoLoRA-MOEAD/
├── pyproject.toml
├── src/
│   └── evolo_moead/
│       ├── __init__.py         # exposes EvoLoRATuner
│       ├── api.py              # high-level orchestrator
│       ├── core/
│       │   ├── gps_metric.py   # SVD gradient projection score
│       │   └── geometry.py     # Pareto knee-point extraction
│       ├── evaluator/
│       │   ├── base.py         # evaluator ABC + MockEvaluator
│       │   └── hf_evaluator.py # HuggingFace gradient evaluator
│       └── search/
│           ├── problem.py      # pymoo multi-objective problem
│           └── optimizer.py    # MOEA/D driver
├── examples/
│   └── quick_start.py
└── README.md

---

Citation

This package is an independent engineering implementation and secondary creation based on the method described in Zero-Shot Evolutionary Architecture Search for Low-Rank Adaptation (Wu et al., 2026). Cite the original work when applying this engineering implementation in research. DOI: 10.1142/S0129065726500255

License

Apache-2.0.