halflife

Your agent works on 5-step tasks. Does it work on 50?

halflife measures your agent's reliability half-life: the task length at which it succeeds half the time. Then it tells you where runs broke, what to change, and re-runs exactly the cases that failed once you've changed it.

Agents fail quietly as tasks get longer: they lose a running total from step 4 by step 30, skip the last item, pay a duplicate they saw twenty steps ago. Unit tests and single-turn evals don't catch it. halflife does, in your terminal, your tests, your CI, and your coding agent.

The numbers above come from halflife demo, a simulated agent built to show the tool. Run it on your own agent for real numbers.

Start in one minute

pip install halflife-ai
halflife demo

Then pick the way in that fits what you have.

You build with Claude Code, Cursor or Codex → let your coding agent use it

Most agents today are built with a coding agent. Give it halflife once and it checks every agent it builds for you:

claude mcp add halflife -- halflife mcp

Then ask it in plain words: "check how reliable my agent is on long tasks and fix what breaks." It runs the benchmark or audits your logs, reads where runs broke and why, proposes a change for you to approve, and replays exactly the failing cases to prove the fix worked. A Claude skill and setup for other editors are in integrations/.

You have agent logs → audit them

No setup, any framework, any model. Point it at your logs (JSON Lines of tool calls) and let halflife find the rules your agent follows:

halflife audit agent_logs.jsonl --suggest-rules

It mines patterns from your own runs (a lookup before every action, one notification per claim, every listed item handled), writes them to rules_suggested.py with how often each held, and audits against them. Rules that held in every run guard future runs. Rules that broke in a few runs point straight at the runs that went wrong. The file describes what your agent usually does, so read it, delete anything that's coincidence, and keep what's policy.

Or write the rules yourself:

from halflife.rules import at_most_once, covers, eventually, requires

RULES = [
    covers("list_claims", field="claims", actions=["finalize_claim", "flag_for_review"], key="claim_id"),
    requires("finalize_claim", before="get_claim", match=["claim_id"]),
    eventually("finalize_claim", then="notify_member", match=["claim_id"]),
    at_most_once("notify_member", key=["claim_id"]),
]

halflife audit agent_logs.jsonl --rules rules.py

You get the half-life measured in tool calls, which rules break and how often, and where. No answer key needed. Try it on the sample: halflife audit examples/traces/claims.jsonl --suggest-rules.

You have agent code → benchmark it

from halflife import benchmark

@benchmark(suite="claims-intake")
def my_agent(task):
    return agent.run(task)

halflife run my_agent.py --quick

Your agent runs against a realistic simulated system with a hidden answer key, on tasks from 5 to 40 steps, several times each. Or scaffold everything with halflife init.

You want to compare models → bench them

halflife bench --model ollama:qwen2.5:7b --model anthropic:claude-sonnet-5-5 --quick

Built-in agents for Ollama, LM Studio, vLLM, OpenAI, OpenRouter, Together, Groq, Anthropic, and any OpenAI-compatible server. No code.

Every run tells you what to do next

• Half-life with a 95% interval. Only lengths you actually tested count: if you tested up to 20 steps, it says "≥ 20", never a made-up number.
• Decay shape. Do errors compound as tasks grow, or happen early? The fix differs.
• Where runs broke. The first failure in each run, grouped: lost track of state, skipped a step, acted without a check, looped, broke a rule.
• What to fix first. Ranked changes, structural fixes before prompt tweaks, each with a failing run as evidence.
• Should it act on its own? --workflow-length 26 predicts success at your real workflow length and recommends act autonomously, act with approval, suggest only, or shadow mode.
• Change since last run. Every run compares itself with the previous one: half-life 32.7 → 28.1 steps (−14%, worse).
• An eval pack. The exact failing tasks, saved. After a fix: halflife run my_agent.py --replay .halflife/<run>.evals.json.
• An HTML report to share, and JSON for your pipeline.

Make it a habit

In your tests:

import halflife

def test_agent_is_reliable():
    halflife.assert_reliable(my_agent, min_half_life=15)

In CI (halflife init --ci writes the workflow):

halflife run my_agent.py --min-half-life 20

In your README: halflife badge writes

Plug in any agent

Your function gets a task:

Full examples: examples/claude_agent.py, examples/openai_agent.py.

OLLAMA_CONTEXT_LENGTH=32768 ollama serve

Rules

Invariants checked on every run's tool calls. They need no answer key, so they work on any agent and on production logs.

Use them in audits, or add them to benchmarks: halflife run my_agent.py --rules rules.py.

Built-in suites

Every suite ships with a test proving that an agent following the written policy exactly scores 100%, so a failure is the agent's, not the answer key's. Write your own suite in about 40 lines: examples/custom_suite.py. halflife show --suite claims-intake --length 10 --answers prints exactly what an agent sees, plus the answer key.

How the number works

Success at length L is modeled as 2^(−(L/h)^k). h (half-life) and k (decay shape) are fit by maximum likelihood over every run, with a bootstrap 95% interval. The observed 50% crossing is reported alongside as a model-free check. Full definitions, file formats, and reporting rules are in SPEC.md, so any tool can produce comparable numbers.

Related work. Measuring capability by the length of task completed half the time builds on METR's time-horizon research on frontier models. halflife applies the idea to your own agent on your own workflows, explains why runs fail, and makes it a daily development check.

Commands

halflife run FILE[:function] [--quick] [--suite S] [--lengths 5,10,20,40] [--tasks N] [--runs N] [--rules FILE] [--workflow-length W] [--min-half-life H] [--replay PACK] [--save-traces FILE] [--workers N] [--out DIR]
halflife bench --model PROVIDER:MODEL [--model ...] [same options]
halflife audit TRACES.jsonl [--rules FILE] [--workflow-length W] [--min-half-life H]
halflife init [--suite S] [--ci]
halflife badge [RESULT.json]
halflife compare RESULT.json [RESULT.json ...]
halflife show [--suite S] [--length N] [--answers] [--json]
halflife mcp
halflife demo [--suite S] [--quick]
halflife suites

Roadmap

• Generate test environments automatically from an MCP server or OpenAPI spec, so any agent gets a benchmark with no hand-written suite
• Read OpenTelemetry GenAI traces directly in halflife audit
• Adapters for popular agent frameworks
• TypeScript SDK
• More suites: support, IT ops, procurement, coding

Ideas and suites welcome: see CONTRIBUTING.md.

License

Apache 2.0