driftless

Poetry-style lock regeneration for prompts — delivered Dependabot-style.

A prompt is pinned to a model and an eval dataset (like pyproject.toml declares deps and poetry.lock pins what works). When either moves, the prompt goes stale. driftless repairs it through your real eval, validates on holdout, and opens a PR with evidence.

Also described as Dependabot for LLM models — same automation shape, different core insight: prompts are lockfiles, not just config files.

Status: early development — 0.3.x release line on PyPI. Upgrading from 0.2.x? Version 0.3.0 rejects legacy migration.allow_* fields; follow the upgrade guide before updating.

Install

pip install driftless

Quickstart

Try Driftless without provider keys by copying the bundled support-classifier example:

driftless copy-example support-classifier --out-dir driftless-classifier-demo
cd driftless-classifier-demo
driftless validate -w support_classifier
driftless compare -w support_classifier --to gpt-4o-mini

The comparison intentionally fails:

F1          current 1.000   target 0.000
Total cost  current 0.024   target 0.004
FAIL min_f1: 0.000 >= 0.9

The target is cheaper, but it is not safe to ship because it fails the classifier's quality gate. Continue through the blocked migration path without provider keys:

driftless migrate -w support_classifier --to gpt-4o-mini --generator none
driftless report -w support_classifier
driftless open-pr -w support_classifier

migrate exits non-zero with BLOCKED, as intended. --generator none makes no repair edits, report renders the saved evidence, and open-pr is a dry run unless you explicitly pass --create.

Product proof

This is the actual output of the cold-install quickstart:

A deterministic offline migration was also run against the public support-classifier-svc testbed. It produced draft PR #4 with the generated scorecard, holdout evidence, prompt diff, and model update:

Other bundled examples are available for retrieval QA and tool-using agents:

driftless copy-example rag-qa
driftless copy-example tool-agent

To adopt Driftless in an existing repository, start there with driftless scan, then run driftless configure <workflow> for a detected workflow before validating and comparing it.

How it works

You describe your model-dependent workflow once in driftless.yml: how to run it, how to override the model, which files may be edited, and what quality thresholds must hold. driftless orchestrates your workflow under different models, compares results, repairs allowed files, validates on holdout, and opens a PR with the evidence.

The customer owns the workflow. The tool orchestrates it.

Not a classifier? Choose a grading mode that fits the task — the same loop then optimizes against it, with your team owning the definition of "good":

• eval.score_field / eval.pass_field — your command emits a numeric score or a pass/fail per record (works for any task: summarization, codegen, agents).
• eval.fields — structured extraction, scored per field with precision/recall/F1 against the gold record.
• eval.judge — an LLM judge grades each free-form output against a rubric (with an optional human-scored calibration set for a judge-agreement check). Run driftless judge-check -w <workflow> before optimizing; set max_mae / min_correlation in the contract to gate migrate / compare.

CLI

Configuring when to migrate

plan reads an optional .driftless/policy.yml — the "dependabot.yml" layer. Scaffold it with driftless init-policy; every field matches a default, so an empty file behaves like no file. It controls which triggers are enabled (deprecation is on and forced; cost/quality/new_model are opportunistic), the thresholds a candidate must clear (min_savings_pct, min_gain), a cooldown_days to skip freshly-released models, candidate allow/deny globs, and an ignore list to snooze specific models or moves. The engine still decides whether a candidate actually passes your eval — policy only decides whether to propose it.

GitHub-native usage

A composite GitHub Action (action.yml) wraps the CLI so scans and migrations can run in CI. See .github/workflows/ for a scheduled deprecation scan, weekly plan --act triage, and manually-triggered migration workflows.

- uses: driftless-dev/driftless@v0.3.0
  with:
    command: scan

Documentation

• Blog series: common use cases — drafts for model migration, dataset refine, CI automation, cost, label audit, and LLM judges.
• Getting started — run the bundled classifier, RAG, and agent examples.
• Upgrading to 0.3 — replace legacy migration.allow_* fields with exact files.editable paths.
• Command chooser — map common user situations to CLI commands.
• Known limits — current boundaries before broad rollout.
• Cost and budget guidance — practical defaults for expensive eval loops.
• Launch check — latest local suite, packaging, and example command results.
• Visual proof plan — checked-in visual excerpts and screenshot targets before public launch.
• Example review artifact — dry-run issue/report from a blocked migration.
• Example successful PR artifact — dry-run PR/report from a passing migration.
• RAG and agent workflows — contract patterns for retrieval QA, judge grading, and tool-using agents.
• User readiness plan — what remains before a broader self-serve launch.
• Release process — changelog, tagging, GitHub Releases, PyPI.
• Changelog — version history.
• Repair prompts & custom generators — customize the LLM repair prompt or plug in your own patch generator.
• Run viewer — inspect optimization attempts, metrics, and diffs.