AutoRAG-Optim

Stop guessing your RAG configuration. Let AutoRAG find the optimal one for your data.

AutoRAG-Optim is a CLI tool that automatically discovers the best RAG (Retrieval-Augmented Generation) hyperparameters for your specific database. Instead of manually testing hundreds of parameter combinations, run one command and get a production-ready configuration optimized for your data.

Why AutoRAG?

Most teams waste weeks manually tuning RAG settings—chunk sizes, embedding models, retrieval counts—without knowing what actually works best for their data. AutoRAG solves this by:

• Generating synthetic test data from your documents (no manual labeling needed)
• Intelligently searching the configuration space (20-30 experiments instead of 1000+)
• Evaluating with real metrics (accuracy, faithfulness, relevancy, recall)
• Running entirely locally (ChromaDB for vectors—no Pinecone API key required)

Typical results: 30-40% cost reduction and 20-35% accuracy improvement over default settings.

⚠️ API Cost Warning
AutoRAG makes many LLM API calls during optimization. Ensure you have sufficient API credits.
Default settings (5 questions, 5 experiments) ≈ 100 API calls. Larger runs can use 1000s of calls.

Features

Installation

pip install autorag-optim

For RAGAS evaluation (optional):

pip install autorag-optim[ragas]

Quick Start

1. Create Configuration

Create a config.yaml file:

database:
  type: supabase
  url: https://your-project.supabase.co
  key: your-supabase-anon-key
  bucket: pdf
  folder: pdf

llm:
  provider: groq
  model: null  # Uses default: llama-3.3-70b-versatile

api_keys:
  groq: your-groq-api-key

rag:
  chunk_size: [256, 512, 1024]
  chunk_overlap: [50, 100]
  embedding_model:
    - all-MiniLM-L6-v2
  top_k: [3, 5, 10]
  temperature: [0.3, 0.7]

optimization:
  strategy: bayesian    # or: grid
  num_experiments: 20
  test_questions: 50

evaluation:
  method: custom        # or: ragas

2. Run Optimization

autorag optimize --config config.yaml

3. View Results

autorag results --show-report

Configuration Options

Optimization Strategy

Evaluation Method

LLM Providers

Database Connectors

Estimated API Calls & Runtime

Understanding the cost before running optimization:

Formula

LLM Calls ≈ Q&A Generation + (Experiments × Questions × Calls per Question)

Where:
- Q&A Generation = ceil(test_questions / 2)  [~1 call per 2 questions]
- Calls per Question = 1 (RAG query) + 3 (evaluation) = 4 calls

Estimates by Configuration

*Time varies based on LLM provider rate limits and response times. Groq is typically fastest.

Cost Saving Tips

• Start with fewer experiments (10-15) to validate your setup
• Use bayesian strategy—it finds good configs with 30-40% fewer experiments than grid search
• Reduce test_questions for initial exploration (20-30 is enough to rank configs)

How It Works

┌─────────────────────────────────────────────────────────────────┐
│  1. CONNECT                                                     │
│     Fetch documents from your database (Supabase/Mongo/PG)      │
├─────────────────────────────────────────────────────────────────┤
│  2. GENERATE                                                    │
│     Create synthetic Q&A pairs from your documents using LLM    │
├─────────────────────────────────────────────────────────────────┤
│  3. OPTIMIZE (Two-Phase)                                        │
│     ┌─────────────────────────────────────────────────────┐     │
│     │ OUTER LOOP: Indexing params (expensive)             │     │
│     │   → chunk_size, chunk_overlap, embedding_model      │     │
│     │   → Requires re-indexing documents                  │     │
│     └─────────────────────────────────────────────────────┘     │
│     ┌─────────────────────────────────────────────────────┐     │
│     │ INNER LOOP: Query params (fast)                     │     │
│     │   → top_k, temperature                              │     │
│     │   → Same index, just different retrieval settings   │     │
│     └─────────────────────────────────────────────────────┘     │
├─────────────────────────────────────────────────────────────────┤
│  4. EVALUATE                                                    │
│     Score each config: relevancy, faithfulness, similarity,     │
│     context recall → weighted aggregate score                   │
├─────────────────────────────────────────────────────────────────┤
│  5. REPORT                                                      │
│     Terminal table + JSON + HTML report with best config        │
└─────────────────────────────────────────────────────────────────┘

CLI Commands

autorag optimize --help

Options:
  -c, --config PATH   Path to config file (default: config.yaml)
  --async             Run optimization in background

Evaluation Metrics

Development

# Clone repository
git clone https://github.com/vatsalpjain/autorag-optim.git
cd autorag-optim

# Install with dev dependencies
uv sync --extra dev

# Run CLI
uv run autorag --help

# Run tests
uv run pytest tests/ -v

Requirements

• Python 3.10+
• LLM API key (Groq, OpenAI, or OpenRouter)
• Database (Supabase, MongoDB, or PostgreSQL)
• No Pinecone required—uses local ChromaDB

Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

License

MIT License - see LICENSE for details.