tokenroi-metric 0.1.1

Difficulty-normalized value-per-token efficiency metric for AI-assisted coding

TokenROI

A difficulty-normalized value-per-token efficiency metric for AI-assisted coding sessions.

Industry went from "tokenmaxxing" (reward token volume) to "tokenminning" (cap spend) without a principled way to measure value-per-token. TokenROI scores how much measurable code value a session produced per dollar of tokens, normalized for task difficulty, in a way that resists gaming by spending more.

Install

# recommended (isolated CLI install; modern macOS/Homebrew Python blocks bare pip — PEP 668):
pipx install "tokenroi-metric[bench]"     # brew install pipx first if needed
# or: uv tool install "tokenroi-metric[bench]"

# or in a venv (quotes matter — zsh eats bare [brackets]):
python3 -m venv .venv && . .venv/bin/activate && pip install "tokenroi-metric[bench]"

# or from source:
python3 -m venv .venv && . .venv/bin/activate && pip install -e .

The package imports as tokenroi and the CLI is tokenroi; [bench] adds what benchmark participants need (pytest + the static-signal tools).

Use the harness

# your local dashboard (also what bare `tokenroi` shows): recent sessions, deduped cost,
# cache share, per-project totals — the COST view (value scores come from bench mode)
tokenroi sessions
tokenroi sessions --by-project --since 7d

# score your own session logs (csv with token counts + the iso signal columns)
tokenroi score --input sessions.csv --output scored.csv

# or generate synthetic sessions with known ground truth and score them
tokenroi demo --tasks 40

# or ingest real Claude Code session logs -> tokens + cache-aware cost
tokenroi ingest --logs "~/.claude/projects/*/*.jsonl"

# or ingest ANY tool's usage export (Claude Code, Codex, Cursor, ...) on one scale
# csv columns: session_id, model, in_tokens, out_tokens
tokenroi ingest-usage --input usage.csv

# developer breakdown: WHY a session got its score + how to improve (not semantic — vs band peers)
tokenroi explain --input sessions.csv --session <id>

Benchmark your friends (bench mode — real data, real value)

Same-task cohort benchmarking: N people solve one task pack with their coding agent, each scores their own run locally (no transcript leaves the machine), the organizer ranks rows. Same task = same difficulty band; the pack's gold tests = real functional value; a fresh work dir = diff-scoped maintainability signals. This is TokenROI end-to-end on real data.

# each participant:
pipx install "tokenroi-metric[bench]"   # see Install above if you don't have pipx
tokenroi bench init logparse ~/bench-logparse
cd ~/bench-logparse            # do the task with your agent, any way you like
tokenroi bench submit --name alice   # -> alice.row.csv (cost + value signals only)

# organizer:
tokenroi bench score rows/*.csv      # -> leaderboard + honesty notes

Visible tests are in the work dir; hidden edge-case tests run at submit time from the installed package (honor system in v1). Cost is deduped and cache-aware, labeled API-equivalent — not what anyone paid.

Reproduce the paper's numbers

pip install ".[test,analysis,signals]" tabulate
python -m pytest -q                  # tests: axioms, dea worked example, e2e ground-truth recovery
python analysis/reproduce_all.py     # regenerate EVERY result in one command -> analysis/results/

# or run individual analyses:
python analysis/run_analysis.py     # full study -> analysis/results/
python analysis/robustness.py       # multi-seed robustness (25 seeds) -> mean +/- CI
python analysis/weighted_baseline.py # the foil: when is the weighted ratio arbitrary?
python analysis/padding_attack.py   # gaming resistance: value-padding attack
pip install ".[analysis]"            # matplotlib + semopy for figures and CFA
python analysis/cfa.py              # confirmatory factor analysis of the ISO structure

Paper

The IEEE-format manuscript (IEEEtran) is in paper/tokenroi.tex with paper/references.bib and figures in paper/figures/ — drop into Overleaf to build. A markdown draft lives in docs/paper-draft.md.

How the metric works

raw signals -> orient + normalize per band -> value vector
cost = tokens priced to dollars (the single input; process waste lands here automatically)

lean (headline):   R_s = (mean(value)/cost) / max_band(mean(value)/cost)      in (0,1]
work (wild logs):  value *= W_eff = lines x entropy x survival   (scope varies -> extensive value;
                   padding is entropy-crushed, deleted junk is survival-crushed)
dea (extension):   factor-analyze ~15 signals -> ~3 factors (+ W_eff) -> CCR + cross-efficiency, per band

• Value is the objective, tool-measurable ISO/IEC 25010 battery — no LLM/semantic judging.
• Difficulty is handled by competing only within difficulty bands.
• Weights in the DEA extension are derived by a linear program, not hand-picked.

Results (synthetic study, 160 sessions/dataset, replicated across 25 seeds)

claim	result (mean, 95% range across seeds)
recovers ground-truth ROI (H4)	Spearman 0.985 [0.973, 0.992] within band
resists gaming (H3)	frugal > wasteful in 100% of tasks, every seed
separates equal-quality sessions by cost (H2)	significant (p<0.05) in 100% of seeds
difficulty normalization works (H5)	|corr w/ difficulty| 0.08 stratified (vs ~0.37 unstratified)
lean ≈ DEA (H6)	Kendall τ 0.925 [0.888, 0.960] (lean often suffices)
factors recover ISO groups (EFA)	F1 maintainability, F2 functional/reliability, F3 security; CFA fit moderate (CFI 0.83)
value-padding resistance	rank gain +0.001 vs +0.051 for a coverage-trusting metric

Layout

src/tokenroi/      metric: cost, signals, lean (+ economic_roi), dea, reduction, bootstrap, stats,
                   synth, cli; adapters (real Claude Code logs), code_signals (static ISO signals)
tests/             axiom + e2e + coverage tests (90% coverage)
analysis/          run_analysis, robustness (25-seed), weighted_baseline, padding_attack, cfa
                   -> results/ (tables, figures, hypotheses.json)
paper/             IEEEtran manuscript + references.bib + figures
docs/              design spec + paper draft
LICENSE            MIT

Limitations

DEA scores are relative to the observed cohort (not absolute); the value model is blind to semantic quality by design; validation here is on synthetic data with known ground truth — the harness ingests real agent logs through the same interface for the live study. See docs/ for the full list.