tokenroi-metric 0.1.0

Difficulty-normalized value-per-token efficiency metric for AI-assisted coding

TokenROI

A difficulty-normalized value-per-token efficiency metric for AI-assisted coding sessions.

Industry went from "tokenmaxxing" (reward token volume) to "tokenminning" (cap spend) without a principled way to measure value-per-token. TokenROI scores how much measurable code value a session produced per dollar of tokens, normalized for task difficulty, in a way that resists gaming by spending more.

Install

python3 -m venv .venv && . .venv/bin/activate
pip install -e .

Use the harness

# score your own session logs (csv with token counts + the iso signal columns)
tokenroi score --input sessions.csv --output scored.csv

# or generate synthetic sessions with known ground truth and score them
tokenroi demo --tasks 40

# or ingest real Claude Code session logs -> tokens + cache-aware cost
tokenroi ingest --logs "~/.claude/projects/*/*.jsonl"

# or ingest ANY tool's usage export (Claude Code, Codex, Cursor, ...) on one scale
# csv columns: session_id, model, in_tokens, out_tokens
tokenroi ingest-usage --input usage.csv

# developer breakdown: WHY a session got its score + how to improve (not semantic — vs band peers)
tokenroi explain --input sessions.csv --session <id>

Benchmark your friends (bench mode — real data, real value)

Same-task cohort benchmarking: N people solve one task pack with their coding agent, each scores their own run locally (no transcript leaves the machine), the organizer ranks rows. Same task = same difficulty band; the pack's gold tests = real functional value; a fresh work dir = diff-scoped maintainability signals. This is TokenROI end-to-end on real data.

# each participant:
tokenroi bench init logparse ~/bench-logparse
cd ~/bench-logparse            # do the task with your agent, any way you like
tokenroi bench submit --name alice   # -> alice.row.csv (cost + value signals only)

# organizer:
tokenroi bench score rows/*.csv      # -> leaderboard + honesty notes

Visible tests are in the work dir; hidden edge-case tests run at submit time from the installed package (honor system in v1). Cost is deduped and cache-aware, labeled API-equivalent — not what anyone paid.

Reproduce the paper's numbers

pip install ".[test,analysis,signals]" tabulate
python -m pytest -q                  # tests: axioms, dea worked example, e2e ground-truth recovery
python analysis/reproduce_all.py     # regenerate EVERY result in one command -> analysis/results/

# or run individual analyses:
python analysis/run_analysis.py     # full study -> analysis/results/
python analysis/robustness.py       # multi-seed robustness (25 seeds) -> mean +/- CI
python analysis/weighted_baseline.py # the foil: when is the weighted ratio arbitrary?
python analysis/padding_attack.py   # gaming resistance: value-padding attack
pip install ".[analysis]"            # matplotlib + semopy for figures and CFA
python analysis/cfa.py              # confirmatory factor analysis of the ISO structure

Paper

The IEEE-format manuscript (IEEEtran) is in paper/tokenroi.tex with paper/references.bib and figures in paper/figures/ — drop into Overleaf to build. A markdown draft lives in docs/paper-draft.md.

How the metric works

raw signals -> orient + normalize per band -> value vector
cost = tokens priced to dollars (the single input; process waste lands here automatically)

lean (headline):   R_s = (mean(value)/cost) / max_band(mean(value)/cost)      in (0,1]
dea (extension):   factor-analyze ~15 signals -> ~3 factors -> CCR + cross-efficiency, per band

• Value is the objective, tool-measurable ISO/IEC 25010 battery — no LLM/semantic judging.
• Difficulty is handled by competing only within difficulty bands.
• Weights in the DEA extension are derived by a linear program, not hand-picked.

Results (synthetic study, 160 sessions/dataset, replicated across 25 seeds)

claim	result (mean, 95% range across seeds)
recovers ground-truth ROI (H4)	Spearman 0.985 [0.973, 0.992] within band
resists gaming (H3)	frugal > wasteful in 100% of tasks, every seed
separates equal-quality sessions by cost (H2)	significant (p<0.05) in 100% of seeds
difficulty normalization works (H5)	|corr w/ difficulty| 0.08 stratified (vs ~0.37 unstratified)
lean ≈ DEA (H6)	Kendall τ 0.925 [0.888, 0.960] (lean often suffices)
factors recover ISO groups (EFA)	F1 maintainability, F2 functional/reliability, F3 security; CFA fit moderate (CFI 0.83)
value-padding resistance	rank gain +0.001 vs +0.051 for a coverage-trusting metric

Layout

src/tokenroi/      metric: cost, signals, lean (+ economic_roi), dea, reduction, bootstrap, stats,
                   synth, cli; adapters (real Claude Code logs), code_signals (static ISO signals)
tests/             axiom + e2e + coverage tests (90% coverage)
analysis/          run_analysis, robustness (25-seed), weighted_baseline, padding_attack, cfa
                   -> results/ (tables, figures, hypotheses.json)
paper/             IEEEtran manuscript + references.bib + figures
docs/              design spec + paper draft
LICENSE            MIT

Limitations

DEA scores are relative to the observed cohort (not absolute); the value model is blind to semantic quality by design; validation here is on synthetic data with known ground truth — the harness ingests real agent logs through the same interface for the live study. See docs/ for the full list.