🧠 leo-prompt-optimizer

leo-prompt-optimizer is a production-grade library and CLI tool that transforms raw prompt drafts into structured, high-performance instructions using a 9-step engineering framework.

Stop "vibes-based" prompting. Use a data-driven approach to optimize, evaluate, and benchmark your prompts across OpenAI, Groq, Anthropic, Gemini, and Mistral.

🌟 Key Features

• ⚡ Lightning Fast: Optimized for high-throughput providers like Groq for near-instant iteration.
• 📊 LLM-as-a-Judge: Built-in G-Eval metrics, hallucination detection, and schema adherence checks.
• 🖥️ Rich CLI: Beautiful terminal reports with side-by-side diffs and performance tables.
• 🧩 XML-Structured Output: Automatically reformats prompts into <role>, <task>, and <instructions> blocks for better LLM steerability.

📦 Installation

pip install leo-prompt-optimizer

🖥️ CLI: The "Pro" Workflow

Optimize a prompt and immediately benchmark it against test cases to see if it actually performs better.

leo-prompt --prompt-file draft.txt \
           --provider-name groq \
           --tests tests.json \
           --model your-model-id

What happens under the hood?

1. Optimization: Your draft is expanded into a structured "System Prompt."
2. Execution: Both the Original and Optimized prompts run against your tests.json.
3. Evaluation: A "Judge" model compares the outputs and generates a performance report.

🤖 Supported Providers

🔧 Python API Usage

Perfect for integrating prompt optimization into your CI/CD pipelines or internal tools.

1. Initialize Provider

from leo_prompt_optimizer import GeminiProvider, AnthropicProvider, OpenAIProvider, MistralProvider, GroqProvider, LeoOptimizer, PromptEvaluator, BatchEvaluator

# Automatically loads API keys from .env (GROQ_API_KEY, OPENAI_API_KEY, etc.)
provider = GroqProvider() # or another provider, you can specify your base_url for OpenAI is you have one as an argument
optimizer = LeoOptimizer(
    provider, 
    default_model="your-optimizer-model-id"
)

2. Optimize a Prompt

draft = "Write a code review for this python function."

# Optimize your prompt draft
optimized = optimizer.optimize(
    draft, # Required - the prompt you want to optimize
    user_input_example, # Optional - user input example
    llm_output_example, # Optional - output wanted example
    top_instruction, # Optional - high level instructions if more specific guidance is needed
    model, # Optional - specify the model you want to use
)
print(optimized)

3. Evaluate on a Single Test Input

from rich.console import Console

console = Console()

# Evaluate the optimized prompt on one input compared to the draft prompt
evaluator = PromptEvaluator(provider, optimizer.env, judge_model="your-judge-model-id")
result = evaluator.evaluate(
    original_prompt=draft,
    optimized_prompt=optimized,
    test_input="def add(a, b): return a + b"
)

# Print the evaluation results in a clean and clear table
console.print(result.to_rich_table())

4. Evaluate on a Batch of Test Inputs

test_cases = [
    "def fib(n): return n if n <= 1 else fib(n-1) + fib(n-2)",
    "def fib(n): a, b = 0, 1\n for _ in range(n): a, b = b, a + b\n return a"
]

batch_evaluator = BatchEvaluator(provider, optimizer.env, judge_model="your-judge-model-id")

batch_result = batch_evaluator.run_batch(
    original_prompt=draft,
    optimized_prompt=optimized,
    test_cases=test_cases
)

# Print the batch evaluation results in a clean and clear table
console.print(batch_result.to_rich_table())

Example tests.json format for the CLI:

[
  "def fib(n): return n if n <= 1 else fib(n-1) + fib(n-2)",
  "def fib(n): a, b = 0, 1\n for _ in range(n): a, b = b, a + b\n return a"
]

🧪 The Evaluation Framework

The library provides objective scores to replace subjective testing:

📘 Optimized Format Example

Your raw drafts are transformed into high-signal instructions:

<role>You are a Senior Python Security Auditor...</role>
<task>Analyze the provided function for SQL injection vulnerabilities...</task>
<instructions>
1. Identify all string-formatting operations.
2. Check for missing parameterized queries...
</instructions>
<output-format>Return a JSON object with 'severity' and 'fix'.</output-format>