driftless

When you change the model behind an LLM app, the prompt that used to work often stops working. Driftless runs your eval on the old model and the new one, repairs only the files you allow, and opens a pull request with the evidence — or blocks the change if quality drops.

You need an eval you can run from the command line. The bundled demo needs no API key. Automatic prompt repair on a real workflow does.

If you know Poetry and Dependabot: driftless.yml is the manifest (model + eval dataset), the prompt is the lockfile, and delivery is a gated PR. LLM behavior is empirical, so Driftless scores candidates on your eval instead of resolving versions.

Status: public alpha — 0.3.x on PyPI. Upgrading from 0.2.x? Follow the upgrade guide first.

Install

pip install driftless

Python 3.10 or newer. No API key for the steps below.

Quickstart

Copy the bundled support-classifier example and run it. -w is the workflow name from driftless.yml.

driftless copy-example support-classifier --out-dir driftless-classifier-demo
cd driftless-classifier-demo
driftless validate -w support_classifier
driftless compare -w support_classifier --to gpt-4o-mini

validate checks that the project is wired correctly. compare runs the current model and the cheaper target on the same tiny eval. You should see something like:

F1          current 1.000   target 0.000
Total cost  current 0.024   target 0.004
FAIL min_f1: 0.000 >= 0.9

Read that last line as: the new model scored 0.000, you required at least 0.9, so the cheaper swap is not safe to ship. That failure is the point of the demo.

Continue without API keys. --generator none makes no prompt edits, so migration stays blocked. --generator fixture applies the known-good patch shipped with this example and can pass.

driftless migrate -w support_classifier --to gpt-4o-mini --generator none
# Expected: BLOCKED, non-zero exit. Run the next commands anyway.

driftless migrate -w support_classifier --to gpt-4o-mini --generator fixture
# Expected: PASS — bundled patch, still no API key.

driftless report -w support_classifier
driftless open-pr -w support_classifier
# Dry run: prints what it would open. Add --create only when you mean it.

This demo has only 4 eval rows. It proves install, gating, and a key-free pass/block loop. It is not production evidence. For a real workflow use a representative eval and --generator llm (needs a provider key). See eval confidence.

Words you'll see

Product proof

This is the actual output of the cold-install quickstart:

A larger offline migration was also run against the public support-classifier-svc testbed. It produced draft PR #4 with the generated scorecard, holdout evidence, prompt diff, and model update:

PR #4 is historical proof of a 290-label testbed run. The published CLI reproduces a passing four-row repair with --generator fixture; regenerating PR #4's exact patch still needs provider-backed --generator llm (or the testbed's own simulator) and may differ.

Other bundled examples:

driftless copy-example support-classifier-live
driftless copy-example rag-qa
driftless copy-example tool-agent

To put Driftless on an existing app, follow the existing-repository walkthrough. Start with scan and configure --apply, then review the draft contract before repair or CI.

How it works

You describe the workflow once in driftless.yml: the command that runs it, how to switch models, which files may be edited, and the quality bar. Driftless runs your command under different models, compares results, repairs only allowed files, checks the winner on holdout data, and opens a PR with the evidence.

You own the workflow. Driftless orchestrates it.

Not a classifier? Pick a grading mode that matches the task:

• eval.score_field / eval.pass_field — your command emits a numeric score or a pass/fail per record (summarization, codegen, agents).
• eval.fields — structured extraction, scored per field against gold labels.
• eval.judge — an LLM grades free-form output against a rubric. Run driftless judge-check -w <workflow> before optimizing.

GitHub Action

A composite GitHub Action wraps the same CLI so scans and migrations can run in CI. After you have a working local contract:

- uses: driftless-dev/driftless@v0.3.6
  with:
    command: scan

See .github/workflows/ in this repo for scheduled scan, weekly plan --act, and manually triggered migration examples.

Documentation

Start here

• Landing page — product overview.
• Hosted docs — install, quickstart, contract, CLI.
• Getting started — golden-path example, then adopt in your repo.
• Command chooser — “I want to do X, which command?”
• Known limits — what Driftless will and will not do.
• Eval confidence — when a pass is trustworthy.
• Cost and budgets — how eval loops spend money.

When you need them

• Upgrading to 0.3 — replace legacy migration.allow_* with files.editable.
• Use-case guides — model migration, dataset refine, CI, cost, labels, judges, RAG, agents.
• RAG and agents — contract patterns.
• Repair prompts and custom generators
• Example blocked issue and example success PR
• Run viewer
• Changelog
• Contributing