Modular Prompt Optimization Framework

A framework for evaluating modular combinations of prompt optimization techniques on LLM hallucination reduction. Designed for systematic experimentation with comprehensive evaluation metrics.

Features

• Hallucination Evaluation: correct/incorrect/abstention responses with precision, recall, F1, and hallucination rates
• Multiple Optimizers: Chain-of-Thought, Chain-of-Verification, Expert Persona, Uncertainty Quantification, and their arbitrary combinations
• Multiple Dataset: Full support for OpenAI's SimpleQA hallucination benchmark dataset, and more
• Multi-LLM Support: OpenAI GPT, Google Gemini, via unified interface
• Automatic Checkpointing: Resume interrupted experiments, progress tracking

Quick Start

Prerequisites

Install uv for fast Python package management:

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
# or: pip install uv

Setup

# Clone and setup environment (uv handles everything automatically)
git clone <repository-url>
cd modular-prompt-optimization
uv sync

# Add API keys to .env file
OPENAI_API_KEY=your-key
GOOGLE_API_KEY=your-key  
SCALEDOWN_API_KEY=your-key

Usage

Run Experiments

# Basic experiment
uv run experiment.py --model=scaledown-gpt-4o --optimizers=cot

# Multiple optimizers
uv run experiment.py --model=scaledown-gpt-4o --optimizers=expert_persona,cot

Evaluate Results

uv run evaluate.py -r results/experiment_results.json -d dataset/simpleqa.json

Example output:

🚨 HALLUCINATION ANALYSIS:
   Hallucination Rate: 0.684 (68.4% of attempted answers)
   Abstention Rate: 0.020 (model says 'I don't know')
   
🎯 CORE PERFORMANCE METRICS:
   Precision: 0.316 (accuracy when attempting answers)
   F1 Score: 0.308

Analysis

cd experiments/
jupyter notebook simpleqa_hallucination_analysis.ipynb

Available Options

Models: scaledown-gpt-4o, gemini2.5_flash_lite, llama2, llama2_70b

Optimizers: cot, cove, expert_persona, uncertainty (combinable with commas)

Architecture

• src/llms.py - LLM provider implementations

• src/prompt_optimizer.py - Modular optimization techniques

• evaluate.py - Enhanced evaluation metrics

• experiments/ - Analysis notebook and results

Evaluation Metrics

The framework provides detailed hallucination analysis:

• Response Classification: Correct, Incorrect (hallucinations), Abstentions
• Core Metrics: Precision, Recall, F1 Score
• Hallucination Metrics: Hallucination rate, abstention rate, calibration metrics
• Interactive Analysis: Inspect specific response types and compare optimizers

Extension

Add new optimizers by extending OPTIMIZER_PROMPTS in src/prompt_optimizer.py.

Testing

# Run unit tests
uv run pytest tests/

# Run with coverage
uv run pytest tests/ --cov=src

# Install dev dependencies
uv sync --group dev

GitHub Actions CI automatically runs tests on Python 3.8+ for all commits and pull requests.

Requirements

• Python 3.8+ (managed automatically by uv)
• uv package manager
• Dependencies are defined in pyproject.toml

All dependencies are automatically managed by uv - no manual pip installs needed!