🚀 Auto-LoRA

The Automated Hyperparameter Optimization Platform for Efficient LLM Fine-Tuning.

Auto-LoRA is a powerful, scientific framework designed to take the guesswork out of Large Language Model (LLM) fine-tuning. By combining Bayesian Optimization (via Optuna) with High-Performance Training Engines (via Unsloth and PEFT), Auto-LoRA automatically identifies the optimal LoRA (Low-Rank Adaptation) configurations for your specific dataset and hardware constraints.

🌟 Key Features

🎯 Intelligent Hyperparameter Tuning

Stop guessing ranks and learning rates. Auto-LoRA uses Optuna to search for the best combination of:

• LoRA Rank (r) and Alpha
• Learning Rate and Scheduler
• Dropout Rates
• Target Modules

⚡ Unsloth Integration

Built-in support for Unsloth, providing:

• 2x–5x faster training speeds.
• 70% less VRAM usage.
• Automatic fallback to standard PEFT if hardware is incompatible.

📊 Scientific Metric Suite

Move beyond simple loss curves. Auto-LoRA generates journal-grade reports including:

• NLP Quality: ROUGE-L, BLEU, and Semantic Similarity (via Sentence-Transformers).
• Inference Efficiency: Tokens Per Second (TPS), Latency (ms).
• Hardware Profile: Peak VRAM usage, System VRAM efficiency.

📈 Dynamic Visualization

Generate stunning HTML dashboards and publication-quality Matplotlib charts with a single command.

🚀 Quick Start

Installation

Standard Installation (Recommended)

pip install auto-lora

From Source (For Developers)

git clone https://github.com/shrey1720/auto-lora.git
cd auto-lora
pip install -e ".[dev]"

Recommended for NVIDIA GPUs

pip install unsloth xformers

🛠 Usage Guide

1. System Health Check

Ensure your GPU and VRAM are ready for training.

auto-lora doctor

2. Basic Training

Train with default settings and automatic tuning.

auto-lora train --model "meta-llama/Llama-3.2-1B" --data "my_dataset.json" --max-trials 5

3. Using Expert Presets

Auto-LoRA comes with pre-configured settings for specific domains:

• --preset chatbot: Optimized for conversational flow.
• --preset coding: Lower learning rate, optimized for logic.
• --preset summarization: Focuses on context retention.

4. Scientific Benchmarking

Compare your trained adapter against ground truth answers to get a technical profile.

auto-lora benchmark --run <run_id> --references test_set.json

📂 Project Architecture

The system is modularly designed for extensibility:

auto_lora/
├── tuner/        # Bayesian optimization and search spaces
├── trainer/      # LoRA/QLoRA engine (Unsloth & PEFT)
├── dataset/      # Dynamic loading and scientific validation
├── hardware/     # VRAM analysis and hardware-aware strategy
├── metrics/      # Scorer engine (NLP & Performance)
├── reports/      # HTML Exporters and Chart Generators
├── db/           # SQLite persistence for all runs/trials
└── cli/          # Typer-powered command interface

🔬 Technical Roadmap

• Multi-GPU Support: DDP and FSDP integration.
• DPO Tuning: Direct Preference Optimization tuning loop.
• Custom Scoring Functions: Allow users to define their own success metrics.
• HuggingFace Hub Integration: Direct upload of tuned adapters.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.