Aletheia-LoRA

Gradient-guided layer selection for efficient LoRA fine-tuning.

Aletheia identifies the most informative transformer layers with a lightweight gradient probe, then applies LoRA adapters only to those layers. Across the finalized Research Line A evidence pack, this yields a 15-28% training speedup with bounded extra forgetting and broadly matched downstream behavior across 14 successful models from 8 architecture families spanning 0.5B to 72B parameters.

Paper: Aletheia: Gradient-Guided Layer Selection for Efficient LoRA Fine-Tuning Across Architectures

DOI: 10.48550/arXiv.2604.15351

Hugging Face card: aletheiaprotocol/aletheia-lora

Evidence dataset: aletheiaprotocol/aletheia-lora-evidence

Installation

Until the package is published to PyPI, install it from source:

git clone https://github.com/aletheiaprotocol-ai/aletheia-lora.git
cd aletheia-lora
python -m pip install -e ".[dev]"

Quick Start

import torch
from aletheia_lora import gradient_probe, select_layers, apply_aletheia_lora
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import get_peft_model

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B")

# 1. Measure per-layer importance with a short gradient probe.
layer_scores = gradient_probe(
    model=model,
    dataset=train_dataset,       # tokenized HF Dataset with input_ids + attention_mask
    num_layers=36,
    probe_batches=5,
    chunk_size=8,
)

# 2. Select the top 50% most informative layers.
selected_layers = select_layers(layer_scores, top_pct=50)

# 3. Apply LoRA only to the selected layers.
peft_config = apply_aletheia_lora(
    model=model,
    selected_layers=selected_layers,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    r=16,
    lora_alpha=32,
)
model = get_peft_model(model, peft_config)

To use the paper-style asymmetric attention/MLP allocation:

peft_config = apply_aletheia_lora(
    model=model,
    selected_layers=selected_layers,
    attention_r=16,
    mlp_r=64,
)

Supported Claim

Use this wording:

Aletheia provides a reproducible training-efficiency improvement over standard LoRA across multiple model families and scales, with bounded extra forgetting on downstream benchmarks.

This package supports that finished narrow claim. It does not claim:

• universal replacement for every LoRA setup
• better quality on every model and every metric
• compression better than quantization
• proof of the full Aletheia or NEXUS paradigm

How It Works

1. Gradient probe: computes per-layer gradient norms on a small probe batch set with bounded memory usage.
2. Layer selection: ranks layers by gradient magnitude and keeps the top-k percent.
3. Selective LoRA: applies standard PEFT LoraConfig only to the selected layers.

Evidence Summary

• 81 experiment rows across 14 successful models and 8 architecture families
• 100% per-model speed win rate in the main speed campaign
• 23.1% mean speedup in the strongest frozen campaign
• Bounded extra forgetting on MMLU, GSM8K, and HumanEval under the narrow-claim framing
• Scale range from 0.5B to 72B parameters, including a MoE model

Repository Contents

Reproducibility

The full empirical record is described in the paper. A sanitized aggregate evidence companion is published at aletheiaprotocol/aletheia-lora-evidence. It contains aggregate metadata and final figures only; it excludes credentials, raw datasets, checkpoints, adapter weights, full gradient score dictionaries, and private cloud paths.

Citation

@misc{saket2026aletheia,
  title={Aletheia: Gradient-Guided Layer Selection for Efficient LoRA Fine-Tuning Across Architectures},
  author={Saket, Abdulmalek},
  year={2026},
  eprint={2604.15351},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  doi={10.48550/arXiv.2604.15351},
  url={https://arxiv.org/abs/2604.15351}
}

License

Apache 2.0