dspy-auto-gepa 0.1.9

About

AutoGEPA automates the tedious parts of setting up a DSPy optimization pipeline: converting raw data into dspy.Examples, generating a metric file with an LLM, splitting datasets, running baselines, and training with GEPA.

• Automatic field inference — Maps row columns to DSPy Signature fields automatically
• LLM-generated metrics — Drafts evaluation metrics automatically, saving them as reproducible .py files for human review
• End-to-end pipeline — Datasets → Baseline → GEPA optimization → Compare & promote
• Zero-config defaults — Sensible defaults for all hyperparameters and LMs

Requires DSPy 3.1+.

(back to top)

Quick Start

Install

Install AutoGEPA with uv (recommended):

uv add dspy-auto-gepa

Or with pip:

pip install dspy-auto-gepa

Basic Usage

import dspy
from dspy_auto_gepa import AutoGEPA

# Configure models
lm = dspy.LM("openrouter/openai/gpt-oss-120b")
large_lm = dspy.LM("openrouter/moonshotai/kimi-k2.5")
dspy.configure(lm=lm)

class TicketSignature(dspy.Signature):
    """Classify support tickets."""
    message: str = dspy.InputField()
    urgency: str = dspy.OutputField()
    sentiment: str = dspy.OutputField()

program = dspy.ChainOfThought(TicketSignature)

rows = [
    {"message": "The server room AC is out and equipment is overheating.", "urgency": "high", "sentiment": "negative"},
    {"message": "Can someone clean conference room B next week?", "urgency": "low", "sentiment": "neutral"},
]

# Fields are automatically inferred from the module's signature
auto = AutoGEPA(
    name="TicketSignature-v1_0_0",
    rows=rows,
    module=program,
    metric_lm=large_lm,
    reflection_lm=large_lm,
)

results = auto.run(force=False)  # Set True to re-run even if a saved model exists

# Check if a cached model was loaded
if results.loaded_from:
    print(f"Loaded existing model from {results.loaded_from}")
else:
    print(f"Baseline score: {results.baseline:.4f}")
    print(f"Optimized score: {results.optimized:.4f}")
    print(f"Improvement: {results.improvement:.4f}")
    print(f"Saved optimized program to {results.saved_to}")

(back to top)

Usage

Automatic field inference (recommended)

When your row columns match your module's signature fields, you don't need to specify any field mappings. AutoGEPA infers input and output fields directly from the dspy.Signature attached to your module.

If your row columns don't match the module's signature fields, AutoGEPA will raise a clear error telling you which fields are missing and suggesting you use dict mappings:

ValueError: Row columns do not match module signature fields. Missing from rows: ['message', 'sentiment', 'urgency']. Extra in rows: ['msg_text', 'sent', 'urg']. Pass input_fields/output_fields to map row columns to signature fields, or ensure row columns match exactly.

Loading datasets from files

Pass a file path directly as rows — format is auto-detected from the extension:

auto = AutoGEPA(
    rows="data/train.jsonl",   # .jsonl, .json, .csv, .parquet all work
    module=program,
    name="TicketSignature",
    metric_lm=large_lm,
    reflection_lm=large_lm,
)

With dict field mappings

When your row columns have different names than your module's signature fields:

# Row columns: msg_text, urg, sent
# Signature fields: message, urgency, sentiment

auto = AutoGEPA(
    rows=rows,
    module=program,
    name="TicketSignature",
    input_fields={"msg_text": "message"},    # row_col → sig_field
    output_fields={"urg": "urgency", "sent": "sentiment"},
    metric_lm=large_lm,
    reflection_lm=large_lm,
)

results = auto.run()

Advanced: step-by-step control

If you prefer fine-grained control over each stage, you can call the individual methods that run() orchestrates under the hood:

# Optional: generate the metric file first for human inspection
metric_file = auto.build_metric()
print(f"Metric written to {metric_file}")
# After reviewing, proceed:

ds = auto.datasets()

baseline = auto.run_baseline(datasets=ds)

optimized = auto.train(datasets=ds)

final = auto.run_baseline(module=optimized, datasets=ds)

# Or compare and promote
comparison = auto.compare(
    optimized_module=optimized,
    datasets=ds,
)
auto.promote(
    optimized_module=optimized,
    destination=auto._run_dir / "optimized_ticket_classifier.json",
)

(back to top)

AutoData

AutoData is a built-in synthetic data generator that creates training datasets using LLMs. It reads your module's dspy.Signature to understand the task, then generates realistic input/output rows — either from scratch or seeded with a few examples.

Two generation modes cover different use cases:

• "split" (default) — generates inputs first, then outputs separately. Works well for classification-style tasks where outputs are constrained (e.g., enum labels).
• "signature" — generates complete rows in one shot using the full signature. Better for complex, tightly-coupled outputs like ReAct reasoning traces.

Quick example

import dspy
from dspy_auto_gepa import AutoData, AutoDataConfig

lm = dspy.LM("openrouter/openai/gpt-oss-120b")
dspy.configure(lm=lm)

class TicketSignature(dspy.Signature):
    """Classify support tickets by urgency and sentiment."""
    message: str = dspy.InputField()
    urgency: str = dspy.OutputField()
    sentiment: str = dspy.OutputField()

program = dspy.ChainOfThought(TicketSignature)

seed_rows = [
    {"message": "The server room AC is out and equipment is overheating.", "urgency": "high", "sentiment": "negative"},
    {"message": "Can someone clean conference room B next week?", "urgency": "low", "sentiment": "neutral"},
    {"message": "Thanks for fixing the VPN, works perfectly now!", "urgency": "medium", "sentiment": "positive"},
]

config = AutoDataConfig(
    n=100,
    generation_mode="split",
    seed_examples=seed_rows,
    output_path=".auto_gepa/TicketSignature/generated/rows.jsonl",
)

gen = AutoData(module=program, data_lm=lm, config=config, name="TicketSignature")
result = gen.generate()

print(f"Generated {result.n_produced} of {result.n_requested} rows")
print(f"Time: {result.generation_time_seconds:.1f}s")
for row in result.rows[:3]:
    print(row)

Feeding generated data into AutoGEPA

The typical workflow is: generate data with AutoData, then pass the rows straight into AutoGEPA for optimization:

from dspy_auto_gepa import AutoData, AutoDataConfig, AutoGEPA

# Step 1: Generate synthetic data
gen = AutoData(module=program, data_lm=lm, config=config, name="TicketSignature")
result = gen.generate()
rows = result.rows

# Step 2: Optimize with AutoGEPA
auto = AutoGEPA(
    name="TicketSignature",
    rows=rows,
    module=program,
    metric_lm=lm,
    reflection_lm=lm,
)
results = auto.run()

Loading seeds from files

Use from_csv or from_json to bootstrap generation from existing data:

gen = AutoData.from_csv("seeds.csv", module=program, data_lm=lm)
result = gen.generate(n=200)

AutoDataConfig reference

GenerationResult

gen.generate() returns a GenerationResult with:

(back to top)

API

Constructor

AutoGEPA(...) accepts all configuration fields directly:

Methods

Result Types

• RunResult — returned by run():
  • baseline: float — score before optimization
  • optimized: float — score after optimization
  • improvement: float — absolute difference
  • saved_to: Path \| None — where the optimized program was saved
  • loaded_from: Path \| None — if a cached model was loaded instead of training
• Datasets — returned by datasets():
  • train: list[dspy.Example]
  • val: list[dspy.Example]
  • test: list[dspy.Example]

Field Resolution Behaviour

AutoGEPA resolves fields in this order:

1. Both provided explicitly (list[str] | dict[str, str]) — uses exactly what you gave it. Lists mean exact column names. Dicts mean {row_column: signature_field} mapping.
2. Neither provided — infers both from the module's DSPy Signature. Raises a clear error if row columns don't match signature fields, listing what's missing and what's extra.
3. Only one provided — if the other can be inferred from module signature or remaining row keys, great. If not, raises an error.

(back to top)

Contributing

Quick workflow:

1. Fork and branch: git checkout -b feature/name
2. Make changes
3. Commit and push
4. Open a Pull Request

(back to top)

License

MIT (as declared in pyproject.toml).