Skip to main content

yardstick-predict

Know how long a coding-agent task will take, and what it will cost, before it runs.

License: Apache 2.0 Python 3.12+

yardstick-predict learns from your own past Claude Code and Codex runs. Give it a prompt and an AI model, and it forecasts the expected seconds, tokens and dollars. It's built for orchestrators: programs that run many coding agents and decide which model runs each task, what runs side by side, and what fits a budget or a usage limit.

$ yardstick-predict predict "Add a login page with tests"

  Add a login page with tests

  claude-opus-5-5   max     4 min   1.8M tokens   $0.92
                    xhigh   3 min   1.2M tokens   $0.61
  gpt-6.1-sol       xhigh   5 min   950k tokens   $0.41

  Expected values from your own runs · predictor default, updated 2 days ago · add --json for details

The numbers above are an example. Yours come from your own history.

Features

  • Expected time, tokens and cost for any task on any AI model and effort level you've used, with the full probability curve on request.
  • Learns from records you already have. Claude Code and Codex keep a record of every run. There's nothing to label.
  • Tests itself before it forecasts. It replays your newest tasks and gives forecasts only for AI models that pass.
  • One fixed, versioned JSON format, with published schemas, so any program in any language can call it.
  • Runs on your computer. Only the text you choose is sent: to Voyage to embed it and, if you turn on LLM analysis, to the AI model you pick.
  • Shows the exact cost before spending anything, and counts Voyage's free tokens.
  • You choose the features, and training uses exactly those. The task, its similar past tasks and the project's recent tasks are always used. You can add the conversation so far, the end of the conversation, skills, public model facts, when the task started, an AI model's read of each task, or your own. A feature check tells you which of them really help your forecasts.
  • Advanced settings for experts: output heads, curve shape, loss, dropout, data split, and other model types.

Install

uv tool install yardstick-predict

This installs the newest release from PyPI, the Python package index. Between releases, the main branch on GitHub may hold work in progress.

You need uv, a fast tool that installs Python programs. yardstick-predict runs on macOS, Linux and Windows, with Python 3.12 or newer. uv fetches that Python if you don't have it, and everything else comes with the install.

Quick start

You need a Voyage API key. Voyage turns each prompt into an embedding: a list of numbers that captures what the text means.

The express way, in one line. It takes every default and agrees to the cost:

export VOYAGE_API_KEY=your-key
yardstick-predict create --express

The guided way. Run it and answer a few questions:

yardstick-predict

It finds your past runs, asks which features to use, shows the exact cost of each Voyage 4 model, checks your key and keeps it in your computer's password store. Before it goes ahead, it says about how long the rest will take and how much disk space your tasks will need. Then it trains and shows how well it forecast your newest tasks. Each long step shows a progress bar with the time left.

When you pick features beyond the three that are always on, setup offers to check which of them actually help. Some features carry no signal for your kind of work, and with few tasks a feature can't show anything. The check tests the model with each feature and without it, shows what helped for each AI model, and turns on only the features that helped. You confirm or change them before it trains. LLM analysis first reads only a sample of your newest tasks. If you keep it after the check, setup shows what reading the rest costs and asks. The check is advice: say no, and setup trains with the features you picked.

Then, before you start a task, ask what it will take:

yardstick-predict predict "Add a login page with tests"

It shows the expected time, tokens and cost on each AI model your predictor covers, at each effort level you often use it at. --model and --effort name them yourself, --predictor picks another predictor, and --json prints the answer programs get. Use it to pick a model, set a time limit, or see what a batch of tasks will cost before you start.

Run yardstick-predict every week or so. It tells you how many new tasks it found, what embedding them costs and about how long a refresh takes, and asks before it refreshes.

Features the model learns from

Setup lists each feature with its options under it. Type a number to turn a feature on or off, or a number and a letter for one option. The first three are always on: they cost nothing and send nothing anywhere.

Feature What it is Default
Task What the prompt means, how it's laid out (list items, file names, links, lines, whether it asks a question), the AI model, the effort level as a scale from low to max, the agent and who started it. Always on
Similar past tasks What the past tasks closest in meaning really took: the closest one, the closest ten and the closest hundred, how much they varied, quick replies and real work apart, and closer, newer tasks counting more. Plus a first guess from the prompt's whole embedding. Always on
Project history The project's last finished tasks, from any conversation: what they took, and how long ago the last one finished. Always on
conversation, Conversation The conversation so far, such as how long its last task took, and how big it has grown. With context, also how close the prompt is to the end of the conversation. It costs nothing and sends nothing anywhere. On
context, Context The last 1,500 characters of the conversation before the prompt, embedded. On
skills, Skills The skills known before the task runs, embedded by their descriptions. Options: named, the skills the prompt invokes, such as a typed /review, and earlier, the skills used earlier in the conversation. Off
benchmarks, Model facts Public facts about each AI model from OpenRouter's model list: prices, context size, longest answer and release date. They are the same for old and new models. Off
time, Time When the task started. Options: trend, the days since 1 January 2020, and clock, the hour of the day and the day of the week, in your computer's time zone. A forecast's task starts the moment you ask. It costs nothing and sends nothing anywhere. Off
llm, LLM analysis An AI model reads each task before it starts and describes the work. Options: kind, the kind of task and its expected size, and description, a few sentences on the work, embedded. Off
your own Any text or number you compute from a task. See Extend it. —

Training uses exactly the features you choose. The conversation tells a lot about a person's sessions and can mislead about a program's one-off runs, so a predictor that mostly forecasts programs' runs may forecast better without conversation. With LLM analysis, training also makes the same model without it, with a test of its own, for the tasks that LLM analysis can't read.

To find out which features help your own forecasts, run the feature check: setup offers it, and yardstick-predict check runs it on a predictor you already have. The Features screen shows each feature's latest verdicts. How it works explains how to read them.

Choose features at setup, or with flags. A feature named alone gets all its options:

yardstick-predict create --yes --features conversation,context,skills:named,llm --llm-provider claude

A predictor keeps its embedding model for good. To change its features, create a new predictor.

Embeddings and LLM answers cost money or time, so each is made once and kept in a feature store that all your predictors share. A second predictor over the same records pays for nothing it already has. yardstick-predict features lists every feature of every predictor: what's stored, and its coverage, the share of the predictor's tasks that have a value for it. yardstick-predict status shows each predictor's coverage too.

yardstick-predict features delete ID deletes a stored feature, by the id the listing shows. First it says which predictors use it and what making it again would cost: an embedding usually comes back from the cache of vectors for free, and LLM answers have to be asked for again. It asks before it deletes anything.

LLM analysis

It sends each task's prompt and the end of the conversation before it to the AI model you pick, and never what the task went on to do. Pay for it one of four ways:

--llm-provider How it's paid Models
anthropic An Anthropic API key, per use. Setup and refresh show the cost first. Claude Opus 5.5 (the default), Sonnet 5.5, Haiku 5.5
openai An OpenAI API key, per use. Setup and refresh show the cost first. GPT 6.1 Sol (the default), GPT 6 Astra, GPT 6 Luna
claude Your Claude subscription, through the Claude Code command. No extra bill, but it counts toward your plan's usage limits. Claude Opus 5.5 (the default), Sonnet 5.5, Haiku 5.5
codex Your ChatGPT subscription, through the Codex command. No extra bill, but it counts toward your plan's usage limits. Codex's own model

With Claude Opus 5.5, it costs about $8 per 1,000 tasks at list price; with Claude Haiku 5.5, about $0.20. Each forecast reads its own task once more. A subscription runs Claude Code or Codex with no tools, none of your settings or hooks, and no saved session, so these calls never show up in the records your predictor learns from. When Claude Code or Codex isn't signed in, setup and refresh say how to sign in, stop asking, and train without LLM analysis for now. The next refresh asks again. An API key needs the providers' own libraries:

uv tool install 'yardstick-predict[llm]@latest'

Forecasts in Claude Code

yardstick-predict claude-code install puts the forecast in Claude Code's status line. When you send a prompt, it shows the forecast for the model and effort level you're on, counts up while the agent works, then shows how long the task really took:

yardstick · 1:12 of ~4 min · ~$1.23

"Took" is the agent's working time, as training counts it: waits for your answers and approvals are left out. When another hook keeps the agent working after it stops, the clock runs on. A background job reporting back continues its task. Scheduled tasks and messages from other sessions are forecast as a program's prompts, as training counts them. Scripted runs, such as claude -p, get no forecast, since they show no status line.

While it's on, every new prompt you send in Claude Code goes to Voyage to be embedded, a few tokens each. Add --here to turn it on only for sessions in the current folder. yardstick-predict claude-code remove turns it off and puts back the status line it replaced. Both keep a copy of Claude Code's settings file next to it, as it was before.

If your Voyage key is only in the VOYAGE_API_KEY environment variable, Claude Code may not see it. Install says so, and offers to keep the key in your computer's password store.

If you installed it with version 0.2.1 or earlier, run yardstick-predict claude-code install again. The older commands let Python load code from the folder you work in, and the status line asks you to update them until you do.

Use it from a program

predict reads a JSON request on standard input and writes a JSON answer. It never asks questions.

echo '{"version": 1, "tasks": [
  {"id": "a", "prompt": "Fix the flaky test", "models": ["claude-opus-5-5", "gpt-6.1-sol"]},
  {"id": "b", "prompt": "Write the release notes", "models": ["claude-opus-5-5"], "context": "We ship on Fridays."}
]}' | yardstick-predict predict

From Python, an orchestrator can pick the cheapest model for a task like this:

import json
import subprocess

request = {
    "version": 1,
    "tasks": [{"id": "login", "prompt": "Add a login page with tests", "models": ["claude-opus-5-5", "gpt-6.1-sol"]}],
}
done = subprocess.run(["yardstick-predict", "predict"], input=json.dumps(request), capture_output=True, text=True)
answer = json.loads(done.stdout)
if "error" in answer:
    raise SystemExit(answer["error"]["message"])

priced = [f for f in answer["forecasts"] if f["usd"] is not None]
cheapest = min(priced, key=lambda f: f["usd"])
print(f"Run it on {cheapest['model']}: about {cheapest['seconds'] / 60:.0f} minutes and ${cheapest['usd']:.2f}.")
  • effort is the effort level the forecast is for: the request's, or the level the model ran at most often.
  • seconds, tokens and usd are expected values: averages, not middles. They add up, so the expected total of ten tasks is the sum of their ten forecasts.
  • tokens counts every kind: new input, cache writes, cache reads and output, including helper agents.
  • usd is the cost at the model's list price.
  • reason appears only when a forecast isn't given. Then the numbers are null and it says why: unknown_model, not_enough_data or not_accurate_enough.
  • "distribution": true adds each forecast's full curve, so a program can work out any chance or any range.

One request can hold up to 1,000 tasks, each on up to 50 models. The format is fixed and versioned: within version 1, fields may be added but are never renamed or removed. Schemas for the request and the answer let a program check every answer. The format explains every field.

How it tests itself

A forecast you can't trust is worse than none, because a program acts on it. So before it gives any forecast, a predictor replays each AI model's newest 30% of tasks in three windows. Each window is forecast by a model trained only on the tasks that finished before it, on every AI model. So a model you used less lately is still tested on its own newest tasks. Then each AI model must pass three checks:

  1. Beat the simple rule. The simple rule gives every task on a model the same forecast: the spread of that model's past tasks. The forecasts must score at least as well on CRPS, the standard score for probability forecasts, within 2%.
  2. Keep its ranges honest. At least 75% of real results must fall inside the 90% range.
  3. Add up. The expected values must add up to between half and twice the real total.

A model that fails a check gets null and a reason. So does an effort level the model never ran at. yardstick-predict status shows the scores and which check failed. Each AI model needs 60 tasks of its own: at least 30 to test on and 30 before them. Much of how long an agent takes is chance, so the predictor's job is to be honest about that spread, and to narrow it where your data allows.

Advanced settings

These are for experts and have sensible defaults. The guided setup never asks about them.

Flag What it sets Default
--model-type The kind of model: xgboost, or one a plugin adds. xgboost
--heads What to forecast: any of seconds, tokens, usd. all three
--head distribution for a full curve, or expected for the expected value alone. distribution
--shape The curve's shape: lognormal or weibull. Only curves have one. lognormal
--loss What training minimizes: log_score or crps for curves, gamma or tweedie for expected values. log_score
--dropout The share of trees dropped each round, from 0 to 0.9. 0
--test-share The share of each AI model's newest tasks held back as the test, and at least 30. 0.3
--test-windows How many windows each AI model's test is split into. 3
--validation-share The share of training tasks that decides when to stop adding trees. 0.15
--half-life Days until a task counts half as much as the newest. 0 counts every task alike. 30
yardstick-predict create --yes --heads seconds,usd --shape weibull --test-windows 5

Extend it

New features may help forecasts more than anything else. Add your own in a plugin: a Python module that registers them.

# my_features.py
from yardstick_predict.features import register_feature

register_feature("words", "number", lambda info: len(info.prompt.split()))
register_feature("project", "text", lambda info: info.project)
yardstick-predict create --yes --plugin my_features --features conversation,context,words,project

--plugin takes a module in the current folder, as here, or the path of a .py file. The predictor remembers where the file is, so the plugin also loads when a forecast runs from another folder, as the status line's do.

A text feature is embedded like the prompt. A number feature goes to the model as it is. Your function gets the task's prompt, context, skills, project, model, thinking level and agent, from past records when training and from the request when forecasting. It also gets when the task started, as info.started_at, in seconds since 1970: a forecast's task starts the moment you ask. A request can also carry an "extra" object for your features, which reaches them as info.extra. Past tasks have no extra, so a feature should also work its value out from the task itself. Training says so when one of your features had no value for any past task.

A plugin can also add a model type with yardstick_predict.model_types.register_model_type(name, train, load), then use it with --model-type. model_types.py describes what a model type provides.

Commands

Command What it does
yardstick-predict Sets up your first predictor, or finds new runs and offers to refresh.
yardstick-predict create [NAME] Creates a predictor. In a terminal it asks for a name first. --express, or --yes, takes every default, asks nothing and agrees to the cost. --archive codex=PATH adds a folder of records.
yardstick-predict refresh Adds new runs, embeds them and retrains.
yardstick-predict predict "TASK" Forecasts a task: a table in a terminal, JSON for programs.
yardstick-predict status Shows each predictor, its features with their coverage, the size of its prepared dataset and how it did on its test. --json for programs.
yardstick-predict list Lists your predictors, one line each: features, how many tasks each learned from, when it was last trained and which AI models get forecasts. --json for programs.
yardstick-predict delete NAME Deletes a predictor: its settings, its dataset of tasks, its prepared dataset and its model. Your records and the feature store stay. In a terminal it says what goes and asks first. --yes for programs, with --json for an answer in JSON.
yardstick-predict check [NAME] Checks which of a predictor's features help its forecasts: it tests the model with each feature and without it. It's advice only and changes nothing. --json for programs.
yardstick-predict features Lists every feature of every predictor: whether the feature store keeps it, with what made it, how big it is and when it changed, or training works it out for free. Shows the share of each predictor's tasks it covers. --json for programs.
yardstick-predict features delete ID Deletes a stored feature. It says which predictors use it and what making it again costs, and asks first. --yes for programs.
yardstick-predict default [NAME] Shows or sets your default predictor: the one commands use when they name none. --predictor NAME picks another for one command.
yardstick-predict claude-code install Shows forecasts in Claude Code's status line. --here for one folder, remove to turn it off.

Update or remove it

To get the newest version:

uv tool upgrade yardstick-predict

It keeps the llm extra if you added it. If you installed 0.3.0 or an earlier version from GitHub, switch to PyPI once with uv tool install 'yardstick-predict@latest', or uv tool install 'yardstick-predict[llm]@latest' if you use LLM analysis with an API key. After that, the upgrade command above gets each new release.

Upgrading keeps your predictors. When a new version reads records more accurately, the next refresh reads them all again, once, and embeds only the texts that changed. When a new version reads new features or trains the model another way, the next refresh trains it again. Until then, a model the previous version trained keeps forecasting as it did. A model from an older version gives no forecasts until it's trained again.

From 0.4 on, going back to an older version is safe too. It reads what it can of a predictor that a newer version saved, and never changes it: it skips any feature it doesn't know, with a note, and it won't refresh that predictor.

To remove one predictor, run yardstick-predict delete NAME. Your records, the feature store and your other predictors stay.

To remove everything, uninstall the program, delete its folder and delete the Voyage key it kept:

uv tool uninstall yardstick-predict
rm -rf ~/.yardstick
uvx keyring del yardstick-predict voyage-api-key

If you set YARDSTICK_HOME, delete that folder instead. On Windows, delete the .yardstick folder in your user folder. If you only ever used VOYAGE_API_KEY, the last line reports that it found no key. That's fine.

Privacy and safety

  • Your records stay on your computer. The dataset, the feature store and the model live in ~/.yardstick, or wherever YARDSTICK_HOME points. The folder is readable only by you, on Windows too.
  • The cache of embeddings holds no text. For each text it keeps a one-way hash and the text's vector, in the same private folder.
  • Only the text you choose is sent, over an encrypted connection: to Voyage to embed it, and with LLM analysis, each task's prompt and the end of its conversation to the AI model you pick.
  • Your Voyage key is read from VOYAGE_API_KEY or kept in your computer's password store. yardstick-predict never writes it to a file of its own.
  • Other downloads are public data, made only when needed: Voyage's tokenizer from Hugging Face, at one fixed version, and OpenRouter's model list if you use benchmarks.
  • Saved models are plain JSON and plain numbers. Loading one runs no code.
  • Plugins are code you choose to run. yardstick-predict only imports the modules you name.

Cost

Voyage gives every account 200 million free tokens. yardstick-predict can't see your balance, so it keeps its own count and asks once whether your account is new.

Before it embeds anything, it shows the exact token count, counted with Voyage's own tokenizer, and the price. If a big history won't fit in your free tokens, it offers to embed only your newest tasks. Each refresh shows the cost of just the new tasks.

A text is paid for once. The same text in many tasks, such as a prompt that says only "continue", is sent once. A text embedded before with the same Voyage model, for any predictor or for a forecast, comes from a cache on your computer and costs nothing.

An account without a payment method gets low limits, such as a few requests and 10,000 tokens a minute. When Voyage says a limit is reached, yardstick-predict waits for it to come back, so a big first setup on such an account can take a while.

Dependencies

Package Why
XGBoost The gradient-boosted trees. Linux and Windows get xgboost-cpu, the same library without the large GPU part.
NumPy The numbers.
tokenizers Counts tokens exactly with Voyage's own tokenizer, so you see the cost before paying it.
keyring Keeps your Voyage key in your computer's password store.
PyArrow Reads and writes the feature store's files, in Parquet, a compact file format for tables.
scikit-learn, on macOS only Brings the OpenMP library XGBoost needs on a Mac, so there's nothing else to install.
anthropic and openai, only with the llm extra The providers' own libraries, for LLM analysis paid with an API key. Not installed unless you ask for them.

Documentation

Contributing

Bug reports, ideas and pull requests are welcome. See the contributing guide and the security policy.

License

Apache License 2.0.

Metadata

Release files for yardstick-predict 0.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for yardstick-predict 0.4.0
File Size Uploaded Provenance Actions
yardstick_predict-0.4.0.tar.gz 711.9 kB With provenance Details for yardstick_predict-0.4.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for yardstick-predict 0.4.0
File Interpreter ABI Platform Actions
yardstick_predict-0.4.0-py3-none-any.whl Python 3 none any Details for yardstick_predict-0.4.0-py3-none-any.whl

Total release size: 925.7 kB

Release files / yardstick_predict-0.4.0.tar.gz

Download URL yardstick_predict-0.4.0.tar.gz
Size 711.9 kB
Tags Source
SHA-256 checksum
How to use checksums
4b83d05e1bf717c7ebe0c6dbf7fa65f213c873dbe196db38f21eb00a3341224b
BLAKE2b-256 checksum
How to use checksums
54ec544d6aa0fb66fd604a71e44d9f2bc085cfe69412439db7c0af19e6782a03
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 23, 2026.

Transparency log

Release files / yardstick_predict-0.4.0-py3-none-any.whl

Download URL yardstick_predict-0.4.0-py3-none-any.whl
Size 213.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a0cc2ce79a6002110d7d66dacfc289342687b6ceff7f200301699ee12095831e
BLAKE2b-256 checksum
How to use checksums
4ff23181ac42f2cf3fb28667dc9a77bb681ff23c4d3e3291f49becebfa118eb4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 23, 2026.

Transparency log

Security for yardstick-predict 0.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for yardstick_predict-0.4.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 23, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page