driftless

Poetry-style lock regeneration for prompts — delivered Dependabot-style.

A prompt is pinned to a model and an eval dataset (like pyproject.toml declares deps and poetry.lock pins what works). When either moves, the prompt goes stale. driftless repairs it through your real eval, validates on holdout, and opens a PR with evidence.

Also described as Dependabot for LLM models — same automation shape, different core insight: prompts are lockfiles, not just config files.

Status: public alpha — 0.3.x release line on PyPI. Upgrading from 0.2.x? Version 0.3.0 rejects legacy migration.allow_* fields; follow the upgrade guide before updating.

Install

pip install driftless

Quickstart

Try Driftless without provider keys by copying the bundled support-classifier example:

driftless copy-example support-classifier --out-dir driftless-classifier-demo
cd driftless-classifier-demo
driftless validate -w support_classifier
driftless compare -w support_classifier --to gpt-4o-mini

The comparison intentionally fails:

F1          current 1.000   target 0.000
Total cost  current 0.024   target 0.004
FAIL min_f1: 0.000 >= 0.9

Smoke-demo warning: this fixture has only 4 rows. It proves packaging, contract execution, metric gating, evidence rendering, and dry-run PR/issue behavior; it does not establish production quality, statistical confidence, provider behavior, or a successful repair. Use a representative eval and real provider credentials before making a shipping decision.

The target is cheaper, but it is not safe to ship because it fails the classifier's quality gate. Continue through the blocked migration path without provider keys:

driftless migrate -w support_classifier --to gpt-4o-mini --generator none
driftless report -w support_classifier
driftless open-pr -w support_classifier

migrate exits non-zero with BLOCKED, as intended. --generator none makes no repair edits, report renders the saved evidence, and open-pr is a dry run unless you explicitly pass --create.

To reproduce a passing repair on the same bundled example, still without provider keys:

driftless migrate -w support_classifier --to gpt-4o-mini --generator fixture
driftless report -w support_classifier
driftless open-pr -w support_classifier

--generator fixture applies the known-good prompt patch shipped for this example. It proves the published CLI can open a passing evidence artifact; it does not replace --generator llm on a real workflow. The four-row set is still too small for production confidence — see eval confidence.

Product proof

This is the actual output of the cold-install quickstart:

A larger offline migration was also run against the public support-classifier-svc testbed. It produced draft PR #4 with the generated scorecard, holdout evidence, prompt diff, and model update:

PR #4 is historical proof of a 290-label testbed run. The published CLI reproduces a passing four-row repair with --generator fixture; regenerating PR #4's exact patch still needs provider-backed --generator llm (or the testbed's own simulator) and may differ.

Other bundled examples are available for retrieval QA and tool-using agents:

driftless copy-example rag-qa
driftless copy-example tool-agent

To adopt Driftless in an existing repository, follow the guided existing-repository walkthrough. It starts with scan and configure --apply, then gives a concrete draft-to-contract example, exact editable-path rules, provider-cost guidance, and safety checks before repair or CI. configure always saves a reviewable draft; --apply also creates or safely appends the workflow to root driftless.yml without rewriting existing comments.

How it works

You describe your model-dependent workflow once in driftless.yml: how to run it, how to override the model, which files may be edited, and what quality thresholds must hold. driftless orchestrates your workflow under different models, compares results, repairs allowed files, validates on holdout, and opens a PR with the evidence.

The customer owns the workflow. The tool orchestrates it.

Not a classifier? Choose a grading mode that fits the task — the same loop then optimizes against it, with your team owning the definition of "good":

• eval.score_field / eval.pass_field — your command emits a numeric score or a pass/fail per record (works for any task: summarization, codegen, agents).
• eval.fields — structured extraction, scored per field with precision/recall/F1 against the gold record.
• eval.judge — an LLM judge grades each free-form output against a rubric (with an optional human-scored calibration set for a judge-agreement check). Run driftless judge-check -w <workflow> before optimizing; set max_mae / min_correlation in the contract to gate migrate / compare.

CLI

Configuring when to migrate

plan reads an optional .driftless/policy.yml — the "dependabot.yml" layer. Scaffold it with driftless init-policy; every field matches a default, so an empty file behaves like no file. It controls which triggers are enabled (deprecation is on and forced; cost/quality/new_model are opportunistic), the thresholds a candidate must clear (min_savings_pct, min_gain), a cooldown_days to skip freshly-released models, candidate allow/deny globs, and an ignore list to snooze specific models or moves. The engine still decides whether a candidate actually passes your eval — policy only decides whether to propose it.

GitHub-native usage

A composite GitHub Action (action.yml) wraps the CLI so scans and migrations can run in CI. See .github/workflows/ for a scheduled deprecation scan, weekly plan --act triage, and manually-triggered migration workflows.

- uses: driftless-dev/driftless@v0.3.4
  with:
    command: scan

Documentation

• Landing page — product overview and public-alpha proof.
• Hosted documentation — installation, adoption path, concepts, and reference.
• Run viewer — inspect optimization attempts, metrics, and diffs.
• Use-case guides — model migration, dataset refine, CI automation, cost, label audit, judges, RAG, and agents.
• Getting started — run the bundled classifier, RAG, and agent examples.
• Upgrading to 0.3 — replace legacy migration.allow_* fields with exact files.editable paths.
• Command chooser — map common user situations to CLI commands.
• Known limits — supported CLI/Action surface and ownership boundaries.
• Eval confidence — when a pass is trustworthy.
• GA criteria — what 1.0 will and will not include.
• Cost and budget guidance — practical defaults for expensive eval loops.
• Launch check — latest local suite, packaging, and example command results.
• Visual proof inventory — genuine captures, provenance, and reproduction notes.
• Example review artifact — dry-run issue/report from a blocked migration.
• Example successful PR artifact — public testbed PR and separate saved fixture.
• RAG and agent workflows — contract patterns for retrieval QA, judge grading, and tool-using agents.
• User readiness plan — current adoption boundaries and wider-launch follow-up.
• Release process — changelog, tagging, GitHub Releases, PyPI.
• Changelog — version history.
• Contributing — local setup, checks, and dependency policy.
• Repair prompts & custom generators — customize the LLM repair prompt or plug in your own patch generator.