promptolution 2.3.0

🚀 What is Promptolution?

Promptolution is a unified, modular framework for prompt optimization built for researchers and advanced practitioners who want full control over their experimental setup. Unlike end-to-end application frameworks with high abstraction, promptolution focuses exclusively on the optimization stage, providing a clean, transparent, and extensible API. It allows for simple prompt optimization for one task up to large-scale reproducible benchmark experiments.

Key Features

• Implementation of many current prompt optimizers out of the box.
• Unified LLM backend supporting API-based models, Local LLMs, and vLLM clusters.
• Built-in response caching to save costs and parallelized inference for speed.
• Detailed logging and token usage tracking for granular post-hoc analysis.

Have a look at our Release Notes for the latest updates to promptolution.

📚 Scientific Publications Powered by Promptolution

• CANTANTE: Optimizing Agentic Systems via Contrastive Credit Attribution — Zehle, 2026. arXiv
• MO-CAPO: Multi-Objective Cost-Aware Prompt Optimization — Büssing et al., 2026. arXiv
• promptolution: A Unified, Modular Framework for Prompt Optimization — Zehle et al., 2026. EACL 2026
• Can Calibration of Positional Encodings Enhance Long Context Utilization? — Zehle & Aßenmacher, 2026. EACL 2026
• Disambiguation-Centric Finetuning Makes Enterprise Tool-Calling LLMs More Realistic and Less Risky — Hathidara et al., 2025. arXiv
• CAPO: Cost-Aware Prompt Optimization — Zehle et al., 2025. AutoML 2025

🔧 Installation and Quickstart

pip install promptolution

For local inference, add [transformers] (HuggingFace) or [vllm] (vLLM serving), or both.

Promptolution offers three ways to optimize prompts, from a one-line call to full manual control.

Quickstart: promptolution.optimize

For a classification task, optimize builds the components for you. Simply provide your data, an LLM and its credentials, and a description of the task and start optimizing. evaluate then scores the resulting prompts on data the optimizer never saw.

import pandas as pd
from promptolution import optimize, evaluate
from promptolution.utils import dev_test_split

# DataFrame with columns "x" (input) and "y" (label)
df = pd.read_csv("your_data.csv")
dev_df, test_df = dev_test_split(df, test_frac=0.2)

task_description = "Classify each sentence as subjective or objective."
prompts = optimize(dev_df, task_description, model_id="gpt-4o-mini", api_key="YOUR_API_KEY")
scores = evaluate(prompts, test_df, task_description, model_id="gpt-4o-mini", api_key="YOUR_API_KEY")
print(scores)  # DataFrame: prompt, score, best first

Optimization and evaluation are separate calls, so you can skip the evaluation entirely, score on several datasets, or bring your own splits. If your data already ships with splits (as HuggingFace datasets usually do), pass them directly instead of calling dev_test_split.

This covers classification only. For judge or reward tasks, or for full control over every component, build them yourself (below).

Full control: build the components

import pandas as pd
from promptolution.llms import APILLM
from promptolution.tasks import ClassificationTask
from promptolution.predictors import MarkerBasedPredictor
from promptolution.optimizers import CAPO
from promptolution.utils import evaluate_prompts, dev_test_split

# DataFrame with columns "x" (input) and "y" (label)
df = pd.read_csv("your_data.csv")
dev_df, test_df = dev_test_split(df, test_frac=0.2)

llm = APILLM(
    model_id="gpt-4o-mini",
    api_url="https://api.openai.com/v1",
    api_key="YOUR_API_KEY",
)
task = ClassificationTask(
    dev_df, task_description="Classify each sentence as subjective or objective."
)
predictor = MarkerBasedPredictor(llm)
optimizer = CAPO(
    predictor=predictor,
    meta_llm=llm,
    task=task,
    initial_prompts=["Classify the text as objective or subjective.", ...],
)

best_prompts = optimizer.optimize(n_steps=10)
test_task = ClassificationTask(test_df, task_description=task.task_description)
scores = evaluate_prompts(best_prompts, test_task, predictor)
print(scores)  # DataFrame: prompt, score, evaluated on the held-out split, best first

The initial_prompts may be omitted, they are then generated from the task_description.

Experiments: config-driven runs & grids (Hydra)

For reproducible research, config-driven runs and experiment grids (from YAML/CLI, locally or on SLURM, with result files and restart), use the promptolution-experiment CLI: see the experiment guide.

Full tutorial: Getting Started notebook · Docs

🧠 Featured Optimizers

🏗 Components

• Task – Manages the dataset, evaluation metrics, and subsampling.
• Predictor – Defines how to extract the answer from the model's response.
• LLM – A unified interface handling inference, token counting, and concurrency.
• Optimizer – The core component that implements the algorithms that refine prompts.

🤝 Contributing

Contributions are welcome! See CONTRIBUTING.md for the workflow, code quality guidelines, and how to run tests.

📄 Citation

If you use Promptolution in your research, please cite:

@inproceedings{zehle2026promptolution,
  title={promptolution: A unified, modular framework for prompt optimization},
  author={Zehle, Tom and Hei{\ss}, Timo and Schlager, Moritz and A{\ss}enmacher, Matthias and Feurer, Matthias},
  booktitle={Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  pages={282--296},
  year={2026}
}

Developed by Timo Heiß, Moritz Schlager, Tom Zehle, and Henri Oberpaur (LMU Munich, MCML, ELLIS, TUM, Uni Freiburg).