🚀 Ellora

The Automated Hyperparameter Optimization Platform for Efficient LLM Fine-Tuning.

Ellora is a powerful, scientific framework designed to take the guesswork out of Large Language Model (LLM) fine-tuning. By combining Bayesian Optimization (via Optuna) with High-Performance Training Engines (via Unsloth and PEFT), Ellora automatically identifies the optimal LoRA (Low-Rank Adaptation) configurations for your specific dataset and hardware constraints.

🌟 Key Features

🎯 Intelligent Hyperparameter Tuning

Stop guessing ranks and learning rates. Ellora uses Optuna to search for the best combination of:

• LoRA Rank (r) and Alpha
• Learning Rate and Scheduler
• Dropout Rates
• Target Modules

⚡ Unsloth Integration

Built-in support for Unsloth, providing:

• 2x–5x faster training speeds.
• 70% less VRAM usage.
• Automatic fallback to standard PEFT if hardware is incompatible.

📊 Scientific Metric Suite

Move beyond simple loss curves. Ellora generates journal-grade reports including:

• NLP Quality: ROUGE-L, BLEU, and Semantic Similarity (via Sentence-Transformers).
• Inference Efficiency: Tokens Per Second (TPS), Latency (ms).
• Hardware Profile: Peak VRAM usage, System VRAM efficiency.

📈 Dynamic Visualization

Generate stunning HTML dashboards and publication-quality Matplotlib charts with a single command.

🚀 Quick Start

Installation

Standard Installation (Recommended)

pip install ellora

From Source (For Developers)

git clone https://github.com/shrey1720/ellora.git
cd ellora
pip install -e ".[dev]"

Recommended for NVIDIA GPUs

pip install unsloth xformers

🛠 Usage Guide

1. System Health Check

Ensure your GPU and VRAM are ready for training.

ellora doctor

2. Basic Training

Train with default settings and automatic tuning.

ellora train --model "meta-llama/Llama-3.2-1B" --data "my_dataset.json" --max-trials 5

3. Using Expert Presets

Ellora comes with pre-configured settings for specific domains:

• --preset chatbot: Optimized for conversational flow.
• --preset coding: Lower learning rate, optimized for logic.
• --preset summarization: Focuses on context retention.

4. Safe Carry-Forward Training

Tune as usual, then continue the best trial for extra epochs from its checkpoint.

ellora train --model "meta-llama/Llama-3.2-1B" --data "my_dataset.json" --max-trials 3 --continue-best-epochs 2

5. Scientific Benchmarking

Compare your trained adapter against ground truth answers to get a technical profile.

ellora benchmark --run <run_id> --references test_set.json

📂 Project Architecture

The system is modularly designed for extensibility:

ellora/
├── tuner/        # Bayesian optimization and search spaces
├── trainer/      # LoRA/QLoRA engine (Unsloth & PEFT)
├── dataset/      # Dynamic loading and scientific validation
├── hardware/     # VRAM analysis and hardware-aware strategy
├── metrics/      # Scorer engine (NLP & Performance)
├── reports/      # HTML Exporters and Chart Generators
├── db/           # SQLite persistence for all runs/trials
└── cli/          # Typer-powered command interface

🛠 Detailed Command Reference

🚀 Training Options (ellora train)

🧩 Key Terminology

• HPO: Hyperparameter Optimization — automatically finding the best settings.
• LoRA: Low-Rank Adaptation — efficient fine-tuning by only training a small adapter.
• Trial: A single training attempt with one hyperparameter set.
• Run: A complete experiment containing multiple trials.
• Adapter: The learned weights, typically saved as a small file.
• Throughput: Training speed measured in Tokens Per Second (TPS).

🔬 Technical Roadmap

• Multi-GPU Support: DDP and FSDP integration.
• DPO Tuning: Direct Preference Optimization tuning loop.
• Custom Scoring Functions: User-defined success metrics.
• HuggingFace Hub Integration: Direct upload of tuned adapters.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.