Token Saver 1.10.0

Token Saver is a local context-optimization layer for AI coding agents. It reduces unnecessary source, tool-output, and always-on context while preserving exact code where the model needs it.

The project is deliberately conservative: smaller context is useful only when the task still succeeds. Token Saver does not claim a universal percentage reduction in task cost. It measures input size, preserves diagnostics, and keeps omitted command output recoverable.

Install

For Claude Code only, the repository now exposes a marketplace:

/plugin marketplace add elyeshkiri/token-saver
/plugin install token-saver@token-saver-tools

Claude shows the command-source bootstrap for approval before running it. The command-source marketplace path requires Claude Code 2.1.229+. Older Claude Code versions can use the pip + setup path below. The generated plugin calls python -m token_saver.entry, so it does not depend on the token-saver console script being on PATH.

For Claude + Cursor + Codex, or explicit project-managed installation:

pip install claude-token-saver
cd /path/to/project
token-saver setup
token-saver doctor

setup auto-detects Claude Code, Cursor, and Codex, writes only Token Saver-owned integration entries, creates a project .token-saver.toml, and is safe to rerun after upgrades as a repair/migration step. Configure hosts explicitly when needed:

token-saver setup . --host claude --host cursor
token-saver setup . --host all

doctor consolidates CLI, project-config, host-integration, repository-index, and available Claude transcript evidence in one health report. Remove only Token Saver-owned entries with:

token-saver uninstall . --host all
token-saver uninstall . --host all --remove-config

Discover commands without opening the README and enable shell completion:

token-saver commands
token-saver completion bash
token-saver completion zsh
token-saver completion fish

The distribution is named claude-token-saver because PyPI rejects token-saver as too similar to an unrelated existing project. The command and the import are unchanged:

	Name
Install	claude-token-saver
Command	token-saver
Import	token_saver

This is an independent project and is not affiliated with or endorsed by Anthropic.

Documentation

Start with the task-oriented docs instead of searching this README:

• 5-minute quickstart
• Worked end-to-end example
• CLI reference
• Machine-readable CLI contracts
• Configuration reference
• Claude Code / Cursor / Codex integrations
• Troubleshooting
• Upgrading and migration
• Architecture
• Benchmarking methodology
• Validation evidence
• Security and privacy
• Contributing

The complete documentation map is docs/README.md.

Safe oversized-prompt ingress

Claude Code's UserPromptSubmit hook can block a prompt or add context, but cannot replace the submitted text. Token Saver therefore does not claim to rewrite a huge prompt before the model.

Instead, an explicit opt-in can stage very large prompts losslessly:

[ingress]
enabled = true
threshold_tokens = 12000
packet_tokens = 1600

When the threshold fires, Token Saver stores the exact prompt locally with a SHA-256 integrity digest and returns a blocking stage id before Claude processes the prompt. Resume with the generated plugin skill:

/token-saver:ingress STAGE_ID

or manually:

token-saver ingress-show STAGE_ID --path .
token-saver ingress-read STAGE_ID --path . --start-line 80 --end-line 140

The packet contains exact bounded head/tail excerpts plus explicit omitted line ranges. There is no silent first-N-words truncation fallback.

Hybrid semantic retrieval

Token Saver 1.10 adds opt-in chunk-level semantic discovery without turning semantic summaries into edit context. Enable it with either spelling:

token-saver pack . --query "where do stale sessions get rejected?" --semantic
token-saver pack . --query "where do stale sessions get rejected?" --embeddings

The flow is:

versioned structural index
        ↓
overlapping source chunks
        ↓
persistent local vectors
        ↓
exact cosine or optional HNSW
        ↓
bounded lexical/vector rank fusion
        ↓
existing symbol/window selector
        ↓
live exact source bytes

Install the one-step semantic extra, then explicitly download the local model once:

pip install 'claude-token-saver[semantic]'
python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('all-MiniLM-L6-v2')"
token-saver semantic-index .
token-saver semantic-status .

At Token Saver runtime the model loader uses local_files_only=True; semantic retrieval does not silently fetch a model from the network. The SQLite index stores vectors plus repository-relative path/line/symbol coordinates, not source text. If hnswlib is unavailable the same vectors use exact cosine scan instead of changing retrieval semantics.

Semantic evidence is deliberately bounded below exact structural authority. A semantically similar chunk can rescue a natural-language candidate with weak lexical overlap, but an exact requested API identity still carries much more weight.

Persistent retrieval cache and optional Rust fastpath

Application-level context building now caches completed packs across processes when the repository content fingerprint and retrieval configuration are identical. Source digests, index version, changed-file state, ranking feedback, working-set state, and budget/query settings are part of cache identity. Embeddings/custom ranking plugins bypass caching until their external state can be fingerprinted safely.

[retrieval]
cache = true
cache_max_entries = 64

Token Saver also has a separately buildable optional PyO3 accelerator under rust/token_saver_fast. Python remains the reference implementation and automatic fallback. Inspect the active backend with:

token-saver fastpath-status

CI builds the Rust wheel and reruns pack/retrieval/context-quality checks with the native backend required before accepting fastpath changes.

The normal claude-token-saver wheel remains pure Python. To try the optional native accelerator from a source checkout:

python -m pip install maturin
python -m pip install ./rust/token_saver_fast
token-saver fastpath-status

If the extension is absent or fails to import, Token Saver automatically uses the Python reference implementation.

Failure-aware tool output and diagnostic Delta

Bash output now goes through a pluggable processor registry rather than one monolithic filter. Format-specific processors currently cover pytest, Jest/Vitest, git log, and npm/pnpm/yarn/bun installs, with a conservative generic fallback. A processor must explicitly opt into failed-command handling; unknown failures pass through unchanged. After every processor, a shared critical-line recovery pass restores omitted error/location lines, and a ratio gate rejects marginal or larger rewrites.

Inspect routing without running a command:

token-saver output-explain "pytest -q"
token-saver output-explain "npm install" --exit-code 1

Replay captured output against explicit preservation and savings contracts:

token-saver output-replay benchmarks/output-quality.example.json

Each case can require exact diagnostic strings, a maximum output-token budget, and a minimum reduction. A failed contract exits non-zero, so the same fixtures can guard CI.

Repeated pytest and Ruff diagnostics can optionally use graph-aware Delta:

export TOKEN_SAVER_DELTA=1

See OUTPUT_OPTIMIZATION.md for the processor contract, failure-routing rules, critical-line recovery, replay manifest schema, Delta state model, and graph-enrichment behavior.

Within one Claude Code session, subsequent runs classify diagnostics as NEW, CHANGED, UNCHANGED, or RESOLVED. New and changed diagnostics are mapped through Token Saver's repository index to the containing symbol and nearby dependency/call-graph edges. Only bounded structured diagnostics are stored in session state; raw command output is not persisted by Delta. Delta replaces the normal compressed output only when the rendered delta is smaller.

Frozen session-efficiency holdout

Version 1.8 adds a dedicated causal benchmark for the 1.7 session layer. It reuses the already-frozen 24 SWE-bench Verified tasks × 3 trials but compares two condition profiles of the same current Token Saver binary:

v1.6 session-behavior baseline
  Token Saver installed
  continuity=off
  cross-turn dedup=off
  waste detection=off

v1.7 session-efficiency treatment
  Token Saver installed
  continuity=on
  cross-turn dedup=on
  waste detection=on

This isolates the session-efficiency bundle from unrelated 1.7 changes such as new output processors. It is a behavioral baseline, not execution of the historical 1.6 package.

Each arm is deliberately split into two fresh Claude sessions: an investigation-only phase, then a real SessionStart:resume boundary, then a fresh implementation phase. The control gets no continuity context; the treatment can receive the structured checkpoint. The benchmark independently derives tool calls, repeated commands, identical-failure retries, duplicate Reads, and token usage from raw transcripts. Token Saver's own efficiency events are used only to prove which mechanisms activated.

Run/resume locally:

token-saver session-holdout \
  benchmarks/session-efficiency-swebench-24.frozen.json \
  --out session-holdout-runs.json \
  --require-publishable

Evaluate already merged/blind-graded evidence without rerunning agents:

token-saver session-holdout-evaluate session-holdout-runs.json \
  --rates benchmarks/claude-sonnet-5-rates-2026-09-19.json \
  --json --require-publishable

The frozen publication gate requires ≥20 tasks, ≥3 trials/task, isolated arm profiles, independent task success, blind response-quality parity, no manual intervention, complete cache-TTL-aware cost evidence, zero session-efficiency events in the control, one forced continuity restore per treatment run, and observed dedup + continuity + waste signals somewhere in the treatment. A cost-per-success claim additionally requires a positive point estimate and a task-cluster 95% confidence interval whose lower bound is above zero.

The checked-in paid workflow represents 144 arm-runs / 288 Claude task phases, plus blind grading. It is manual/explicitly confirmed. No session- efficiency savings percentage is claimed until that workflow is actually run and passes its publication gate.

Session efficiency: preserve work, not conversation

Token Saver 1.7 adds a host-neutral session-efficiency layer around the existing repository/retrieval and output pipelines.

With Claude Code project hooks installed it now:

• restores a compact structured continuity checkpoint after resume/compaction;
• collapses exact repeated Bash output for the same command and blocks unchanged repeated full-file Reads;
• detects bounded retry loops, repeated commands, and long no-edit tool cascades;
• tracks working files plus recent test/lint/typecheck/build outcomes without storing raw prompt or assistant text;
• records accepted tool-context reductions in a private local event ledger.

Inspect the current working checkpoint:

token-saver continuity .
token-saver continuity . --json

Inspect local efficiency telemetry in the terminal/JSON or generate a dependency-free HTML dashboard:

token-saver dashboard . --days 7
token-saver dashboard . --json
token-saver dashboard . --html .token-saver-dashboard.html

The dashboard deliberately separates exact available Claude usage counters from estimated before/after tool-context tokens saved. It does not turn those operational estimates into a cost-per-success or quality claim; the frozen evidence-run pipeline remains the publication-grade surface.

Command compression also expands beyond pytest/Jest/git-log/package installs to git status, grep/ripgrep/find, Ruff/ESLint/Pylint/Clippy, tsc/mypy/pyright, Go/Cargo tests, common build systems, Python package installs, and Docker/Kubernetes logs. Unknown failures still pass through conservatively and critical-diagnostic recovery remains registry-wide.

Durable project knowledge: reuse conclusions, not just context

Token Saver can now keep explicit, evidence-backed findings across sessions without turning session continuity into a transcript memory system. A finding must include a claim, concrete evidence, an applicability rule, and at least one current repository file anchor:

token-saver remember . \
  --claim "Session refresh is implemented in the auth service" \
  --anchor src/auth.py::refresh_session \
  --evidence "refresh_session delegates the rotation path" \
  --applicability "Use when changing login or refresh behavior"

token-saver recall . --query "debug session refresh"
token-saver knowledge-status .

Each anchor stores the source-file digest that existed when the finding was recorded. If that file changes or disappears, the finding becomes stale and ordinary recall excludes it. New findings may explicitly supersede older ones, and exact claim/anchor duplicates update one record instead of multiplying context. Storage is local, private, project-scoped, bounded, and contains only the finding fields the caller explicitly submits.

This first layer is deliberately conservative: findings are not automatically generated from model conversation and are not silently injected into every context pack. CLI/MCP callers explicitly write and recall them, which keeps the existing retrieval holdouts unchanged while creating a measurable path to future cross-session read/reasoning avoidance.

The same surface is available through MCP as remember_finding, recall_findings, and knowledge_status.

Progressive MCP tool disclosure

Token Saver can advertise a smaller MCP schema when a host needs only its common repository-context operations:

TOKEN_SAVER_MCP_PROFILE=minimal token-saver serve .
TOKEN_SAVER_MCP_PROFILE=context token-saver serve .
TOKEN_SAVER_MCP_PROFILE=full token-saver serve .

full remains the default for backward compatibility. minimal exposes the high-frequency context + durable-knowledge tools; context adds ranking, impact, feedback, index, and knowledge-status operations while omitting diff and output-specialist schemas. Unknown profile names fail closed instead of silently selecting another surface.

Knowledge-assisted read avoidance and cache economics

The durable finding store can now participate in the source-read guard, but only behind an explicit opt-in:

TOKEN_SAVER_KNOWLEDGE_READ_AVOIDANCE=1 claude
TOKEN_SAVER_KNOWLEDGE_READ_AVOIDANCE=1 \
TOKEN_SAVER_CACHE_ECONOMICS=1 claude

For an unbounded source Read, Token Saver checks for verified, active findings anchored to that exact file. A changed/missing anchor, a probable or speculative finding, an allowlisted/non-source file, or a bounded range never qualifies. The replacement must be materially smaller than the file and tells the agent to request an exact offset+limit range whenever implementation bytes are needed.

The optional cache-economics gate evaluates projected relative input cost rather than assuming that fewer raw tokens are always cheaper. It models an already cached prefix separately from the new frontier and charges prefix recreation when a proposed transformation invalidates cached history:

token-saver cache-economics \
  --original-frontier-tokens 4000 \
  --replacement-frontier-tokens 800 \
  --cached-prefix-tokens 12000 \
  --invalidates-cached-prefix \
  --expected-reuses 2

The default 1.25 cache-write and 0.10 cache-read factors are planning defaults, not universal provider pricing. Override them for the active provider/model.

Frozen knowledge-efficiency holdout

The feature is not enabled by default and no end-to-end savings percentage is claimed yet. A separate frozen causal experiment reuses the 24 SWE-bench Verified tasks at three trials per task. Both arms run the same current Token Saver binary, disable continuity/dedup/waste features, perform the same investigation phase, and explicitly persist verified findings. A fresh implementation session then compares memory-control against knowledge-assisted read avoidance plus the cache-economics gate.

token-saver knowledge-holdout \
  benchmarks/knowledge-efficiency-swebench-24.frozen.json \
  --out knowledge-holdout-runs.json \
  --require-publishable

Publication requires independent task success, blind response-quality parity, complete cache-TTL-aware pricing, verified knowledge seeding in every arm-run, zero control read-avoidance activation, observed treatment activation, and a strictly positive task-cluster 95% confidence interval for cost-per-success reduction.

Output Saver: reduce generated tokens too

Token Saver can now control the other side of the bill: model output. The output layer is deliberately split into generation-time policy and safe post-generation compaction.

With Claude Code hooks installed, Token Saver now applies this policy automatically at UserPromptSubmit. It deterministically classifies strong coding, debugging, review, planning, and explanation prompts, while ambiguous follow-ups inherit the active task. The full policy is injected only when the task or verbosity mode changes, on the first task in a session, or after a clear/compact reset, avoiding repetitive policy-token overhead.

Manual policy generation remains available for orchestrators and hosts without a prompt-hook surface:

token-saver output-policy --mode terse --task coding
token-saver output-policy --mode normal --task debugging --max-tokens 700 --json

The policy tells the agent to lead with the useful result, avoid conversational preambles, task restatement, tool narration, repeated logs/context, tangents, unchanged full-file reproduction, verbose test output, recaps, and closing filler. --task adapts both wording and default budgets for coding, debugging, review, explanation, and planning while preserving the historical 300/800/2,000-token defaults when no task is selected.

Debugging policy explicitly separates observations from hypotheses so brevity does not pressure the model into inventing a root cause. Explicit user output contracts, required code/diffs, diagnostics, safety information, and material caveats always override the token target. Explicit requests such as "briefly" or "in detail" also override the configured automatic verbosity for that task. Automatic classification stores only resolved policy metadata (task/mode/budget), not the user's prompt text.

Automatic budgets are adaptive by default. Small/simple tasks can receive less than the static task budget, while multi-part, repository-wide, code-heavy, or diagnostic-heavy prompts receive more room within hard mode-specific bounds. Ambiguous follow-ups keep the active budget unchanged.

Token Saver can also learn safer task/mode bases from real paired experiments:

token-saver output-calibrate benchmarks/agent-runs.json \
  --out .token-saver.output-calibration.json

Calibration accepts only blinded paired evidence, ignores failed/blocked or quality-regressing Token Saver runs, and requires at least three valid samples from three distinct task IDs for a task/mode recommendation. The automatic hook consumes that artifact on future tasks; absent or invalid calibration falls back to built-in defaults.

Claude Code can also record real turn-level usage automatically:

token-saver output-telemetry .
token-saver output-telemetry . --json

The prompt hook checkpoints the transcript byte offset and active policy. At Stop or StopFailure, Token Saver reads only transcript bytes appended for that turn and records input/cache/output token counters, model calls, selected budget, task/mode, and adaptive/calibration metadata. It does not copy prompt text, assistant text, tool payloads, or transcript content. Telemetry reports budget-utilization patterns but deliberately does not treat a finished model turn as verified task success or response quality.

For a complete frozen evaluation, one resumable command now runs the full evidence chain:

token-saver evidence-run benchmarks/e2e-swebench-24.frozen.json \
  --out benchmark-runs.json \
  --require-publishable

It performs randomized baseline/Token Saver trials, independent hidden verification, deterministic blind A/B grading, cache-TTL-aware cost-per-success analysis, and quality-gated adaptive-budget calibration. The shipped frozen suite contains 24 SWE-bench Verified tasks × 3 paired trials (144 agent runs). The GitHub workflow remains explicitly paid/manual and must be executed before any new savings percentage is claimed.

For paired experiments with independent verification and blind response grades, join all four evidence layers:

token-saver output-effectiveness benchmark-runs.json \
  --fresh-input-per-million <rate> \
  --cache-creation-5m-per-million <rate> \
  --cache-creation-1h-per-million <rate> \
  --cache-creation-unknown-per-million <rate> \
  --cache-read-per-million <rate> \
  --output-per-million <rate> \
  --require-publishable

The report measures cost per successful task, checks blind quality parity, verifies policy telemetry against the copied transcript, clusters confidence intervals by task, and identifies task/mode/budget cohorts that have enough quality-preserving cross-task evidence to be candidates for calibration.

Compact an already-generated response:

cat response.md | token-saver output-save --mode terse
token-saver output-save response.md --max-tokens 500 --enforce-budget --json

Safe compaction removes exact repeated prose/status echoes and trivial filler. Fenced code and diffs are preserved byte-for-byte. --enforce-budget may trim prose, but never truncates a fenced code/diff block; if preserved code alone cannot fit, the result reports budget_exceeded: true instead of corrupting the answer.

For agent-to-agent state, compact JSON avoids prose and pretty-print overhead:

cat result.json | token-saver output-save --structured --mode terse

The same capabilities are exposed through MCP as output_policy and compact_output. This lets an orchestrator inject the response policy before the model generates tokens, which is the primary savings path; post-processing cannot refund tokens that were already generated.

Typo-tolerant retrieval and context browser

Token Saver now has a conservative typo/fuzzy layer designed to complement, not replace, structural retrieval.

Repository-level typo correction compares query words only against indexed identifier vocabulary, uses strict similarity and ambiguity margins, and keeps the original terms. Once a file has already survived retrieval, a slightly broader fuzzy fallback can rescue a misspelled symbol identifier without turning fuzzy similarity into a repository-wide ranking signal.

Inspect what the real packer is considering:

token-saver browse . --query "rendr template"
token-saver browse . --query "refresh sesion token" --show 1
token-saver browse . --query "cookie persistence" --interactive
token-saver browse . --query "redirect request" --json

Interactive mode supports list, show N, and quit. The browser reuses the same file ranker, graph evidence, exact source-window selection, secret redaction, and source-visible symbol accounting as pack; it is not a second search engine. It also surfaces any high-confidence fuzzy corrections so a human or agent can see why a typo matched an identifier.

The same inspection surface is available through MCP as browse_context.

Explain ranking score decisions

Ranking observability is opt-in so ordinary packing does not pay for trace collection. Ask for a stage-by-stage breakdown when diagnosing a surprising candidate:

token-saver ranking-explain . --query "refresh session token"
token-saver ranking-explain . --query "refresh session token" --max-files 3 --json

Each candidate reports its final score plus exact score transitions such as BM25, path/symbol evidence, structural authority, file-priority adjustments, changed/working-set/feedback boosts, graph closure, embeddings, and custom registered rerankers. Legacy reasons remain available unchanged.

The same structured payload is exposed through MCP as explain_ranking. Third-party RankingStage implementations are traced automatically when an explanation is requested; plugins do not need their own observability API.

Diff ranking behavior across commits/configurations

Capture the same frozen task manifest on each revision or configuration:

token-saver ranking-snapshot benchmarks/context-quality.json \
  --path . --max-files 20 --out baseline-ranking.json

# after checking out or configuring the candidate
token-saver ranking-snapshot benchmarks/context-quality.json \
  --path . --max-files 20 --out candidate-ranking.json

# configuration experiments are captured in snapshot metadata too
token-saver ranking-snapshot benchmarks/context-quality.json \
  --path . --graph-hops 2 --closure-items 30 --out graph-v2-ranking.json

Then compare the artifacts:

token-saver ranking-diff baseline-ranking.json candidate-ranking.json
token-saver ranking-diff baseline-ranking.json candidate-ranking.json --json

The diff follows every expected file from the manifest, even when it falls below the displayed top-N, and reports rank movement, score movement, and the per-stage contribution changes that caused it. Snapshots with different ground-truth hashes are rejected instead of producing misleading comparisons.

For CI, fail when an expected file disappears or drops farther than an allowed amount:

token-saver ranking-diff baseline-ranking.json candidate-ranking.json \
  --fail-on-regression --allowed-rank-drop 1

This workflow intentionally separates snapshot capture from comparison, so the same diff engine works across Git commits, feature flags, plugin registries, model/embedding availability, or CI artifacts without managing hidden worktrees.

Pull requests run this comparison automatically against the protected base SHA. CI checks out the base and candidate separately, uses the base manifest for both snapshots, writes the Markdown comparison to the GitHub Actions job summary, and uploads baseline-ranking.json, candidate-ranking.json, and ranking-diff.json as a 14-day artifact.

Rank movement is intentionally informational for now: broken snapshot/diff execution fails CI, but ranking regressions do not block merges until a data-backed rank-drop threshold has been calibrated from real PR history.

Calibrate a blocking policy from PR history

Each PR artifact now carries its pull-request number, workflow run/attempt, base SHA, and candidate SHA. A separate Ranking Calibration workflow runs weekly and on manual dispatch. It downloads the newest ranking-regression-<PR> artifact per PR, so workflow reruns do not masquerade as independent evidence.

The workflow groups reports by frozen ground-truth hash and evaluates only the current benchmarks/context-quality.json cohort. Old benchmark definitions are kept separate instead of contaminating the current policy.

You can run the same calibration locally:

token-saver ranking-calibrate ./ranking-history \
  --ground-truth-sha <CURRENT_HASH> \
  --min-reports 20 \
  --markdown

The report includes report/observation counts, expected-file regressions, disappearances, empirical positive rank-drop percentiles, and scoring-stage activity. It also states whether the available history factually supports a zero-rank-drop and/or no-disappearance policy after the configured minimum sample count.

Calibration is deliberately descriptive: it does not call historical regressions "noise" or automatically choose an allowed rank drop. That decision remains a separate ratchet once enough representative PR history exists.

Benchmark Output Saver compaction

Measure the deterministic post-generation layer with a reusable manifest:

token-saver output-benchmark benchmarks/output-saver.json

Each case can provide inline text or a response path, plus its response mode, token budget, budget enforcement setting, and strings that must survive compaction. The report measures original/output tokens, weighted reduction, code-fence preservation, required-content preservation, and budget overflow.

This benchmark deliberately measures only deterministic post-generation compaction. Generation-time policy savings must be measured from actual model runs and can be compared with token-saver cost-report.

Run broad end-to-end cost experiments

For publishable cost-per-success evidence, Token Saver can execute frozen paired coding experiments rather than relying on a handful of manually recorded runs. The harness uses pinned detached worktrees, randomized baseline/enabled order, independent verifier commands, repeated trials, real Claude Code transcripts, and resumable checkpoints.

# freeze task/design definition before paid runs
token-saver experiment benchmarks/e2e-suite.json --print-task-definition-hash

# inspect the randomized 20-50 task schedule without calling a model
token-saver experiment benchmarks/e2e-suite.json --out benchmark-runs.json --dry-run

# execute, then require the broad-evidence protocol in analysis
token-saver experiment benchmarks/e2e-suite.json --out benchmark-runs.json
token-saver benchmark benchmark-runs.json --rates rates.json --require-publishable

The publication gate requires at least 20 distinct tasks and 3 paired trials per task. Cost/success confidence intervals are bootstrapped by task, so repeated trials of one task do not inflate the effective sample size. See BENCHMARKING.md for the frozen-suite protocol and runner schema.

Measure cost per successful task

Context reduction is not the same thing as invoice reduction. Token Saver can compare paired baseline and optimized agent runs directly:

token-saver cost-report baseline.json token-saver.json
token-saver cost-report baseline.json token-saver.json --json

Each run records a task_id, success outcome, input/output/cache tokens, tool/model calls, latency, and optionally cost_usd. If the provider bill is not already available, pass token pricing instead:

token-saver cost-report baseline.json token-saver.json \
  --input-per-million 10 \
  --output-per-million 30 \
  --cached-input-per-million 2

The report computes total cost, success rate, cost per successful task, token/cost/latency reductions, call-count changes, and tasks whose outcome improved or regressed. By default the two files must contain exactly the same task IDs so cost comparisons cannot silently use different workloads.

The same paired manifest already used by agent-evaluate can be passed directly as a single argument, so correctness and economics stay attached to one experiment artifact:

token-saver agent-evaluate benchmarks/agent-runs.json
token-saver cost-report benchmarks/agent-runs.json

Paired manifests use task plus condition: baseline|token-saver; seconds is accepted as latency and is normalized to milliseconds in the cost report.

Release history

Release-specific changes live in CHANGELOG.md. Validation results and evidence limitations live in VALIDATION.md, so this README stays focused on current usage rather than duplicating historical release notes.

Automatic protections

The recommended Claude Code lifecycle is:

token-saver setup /path/to/project --host claude
token-saver doctor /path/to/project

This configures both project hooks and MCP. The lower-level token-saver install command remains available for compatibility and specialized Claude-only/user-wide hook installation; new projects should normally use setup.

Bounded source reads

A full Read of a large source file is denied before it enters context. So is a lone cat <large source file> through Bash, which is the same dump by another route; pipes, redirects, chains and globs are left alone. The denial contains a capped structural outline with line-number gutters so the agent can request an exact range instead.

This happens before the read rather than rewriting its result: editing tools need the original source bytes.

large full read
    ↓
PreToolUse guard
    ↓
outline + exact ranges
    ↓
bounded Read

Duplicate-read blocking is optional and off by default.

Recoverable command-output compression

Large successful Bash output is compressed only when the replacement is materially smaller. Token Saver now also collapses long runs of identical successful log lines while retaining the repetition count.

Errors, exceptions, tracebacks, failed assertions, interrupted commands, images, unsupported structured responses, and stderr are treated conservatively. Test failures preserve diagnostic evidence.

Before replacing output, Token Saver stores the original result locally. The model receives a retrieval command instead of being forced to rerun the command:

token-saver output OUTPUT_ID --stream stdout --offset 1 --limit 80

Prune old originals explicitly:

token-saver outputs-prune --days 7

Source navigation

# signatures/structure of one file
token-saver outline src/service.ts

# exact body of one symbol
token-saver snippet src/service.ts Service.fetchUser

# bounded structural map of a repo
token-saver map src --max-tokens 4000 -o CODEMAP.md

# task-aware working set
token-saver pack . -q "implement retry backoff for downloads" --max-tokens 5000

Python is parsed with ast. JavaScript, JSX, TypeScript, and TSX use Tree-sitter for structural extraction. Other supported languages retain conservative pattern-based outlines.

Measure before claiming savings

Token Saver separates observed API usage from offline estimates:

token-saver sessions /path/to/project
token-saver policy /path/to/project
token-saver audit /path/to/project
token-saver budget /path/to/project
token-saver check /path/to/project

Reports distinguish recorded usage, cache reads/writes, repeated reads, estimated tool-result size, always-on instructions, MCP schema cost, and hypothetical one-shot outline reductions.

For paired real-task benchmarking:

token-saver benchmark runs.json --rates rates.json

See BENCHMARKING.md. A smaller prompt or tool result is not automatically a cheaper successful task; follow-up reads, retries, model quality, and cache behavior all matter.

Context audit and MCP cost

token-saver audit .
token-saver audit . --probe-mcp
token-saver mcp-prune .

The audit identifies instructions and rules that are injected repeatedly, and can measure MCP tool-schema payloads. mcp-prune is dry-run by default.

Standalone filtering

npm test 2>&1 | token-saver filter --command "npm test"
pytest -q 2>&1 | token-saver filter --command "pytest -q"

The filter removes ANSI noise, compacts valid JSON, abbreviates huge hex blobs, collapses duplicate successful log lines, and applies command-aware reductions. Failure evidence is favored over aggressive compression.

Configuration

token-saver setup creates .token-saver.toml in the project. Hook and guard settings can be committed with the repository instead of being repeated as shell environment variables:

version = 1

[hooks]
guard = true
read_max_lines = 220
reread = false
delta = false
min_lines = 40
keep_tail = 15
allow = []

Existing TOKEN_SAVER_* environment variables remain supported and take precedence over project configuration, which keeps CI/temporary overrides simple.

Variable	Default	Purpose
TOKEN_SAVER_GUARD	1	disable with 0
TOKEN_SAVER_READ_MAX_LINES	220	maximum guarded source-read window
TOKEN_SAVER_REREAD	0	optional duplicate full-read denial
TOKEN_SAVER_ALLOW	empty	colon-separated source allowlist globs
TOKEN_SAVER_MIN_LINES	40	minimum Bash stdout lines considered for filtering
TOKEN_SAVER_MAX_LINES	adaptive	filtered output line target
TOKEN_SAVER_KEEP_TAIL	15	tail retained by generic filtering
TOKEN_SAVER_DELTA	0	opt-in graph-aware pytest/Ruff diagnostic Delta
TOKEN_SAVER_INGRESS_OPTIMIZER	0	opt-in lossless oversized-prompt staging before Claude processing
TOKEN_SAVER_INGRESS_THRESHOLD_TOKENS	12000	estimated prompt threshold for ingress staging
TOKEN_SAVER_INGRESS_PACKET_TOKENS	1600	bounded staged packet target
TOKEN_SAVER_RETRIEVAL_CACHE	1	persistent content-fingerprinted completed-pack cache
TOKEN_SAVER_RETRIEVAL_CACHE_MAX_ENTRIES	64	bounded cache entries per project
TOKEN_SAVER_RUST_FASTPATH	1	use optional native extension when installed; 0 forces Python
TOKEN_SAVER_KNOWLEDGE_READ_AVOIDANCE	0	opt-in verified-knowledge replacement for redundant full-file Reads
TOKEN_SAVER_CACHE_ECONOMICS	0	require cache-aware projected-cost approval for knowledge read avoidance
TOKEN_SAVER_CACHE_EXPECTED_REUSES	2	expected future cache reads in the planning model
TOKEN_SAVER_CACHE_WRITE_FACTOR	1.25	relative cache-write input factor; provider/model override recommended
TOKEN_SAVER_CACHE_READ_FACTOR	0.10	relative cache-read input factor; provider/model override recommended
TOKEN_SAVER_CACHE_MIN_RELATIVE_SAVINGS	0.05	minimum projected relative savings for the runtime cache gate
TOKEN_SAVER_CACHE_TTL_MIN	5	advisory cache-gap classification only
TOKEN_SAVER_STATE_DIR	~/.claude/token-saver	local state and recoverable output storage

Design principles

Token Saver follows six rules:

1. Select before compressing. The cheapest irrelevant context is context never loaded.
2. Structure before bodies. Outlines locate the small exact ranges worth reading.
3. Exact bytes for edits. Source windows are never semantic summaries.
4. Failures are evidence. Diagnostics are preserved rather than optimized away.
5. Compression must be recoverable. Omitted command output is stored locally.
6. Measure task outcomes. Token counts alone are not proof of end-to-end savings.

Development

python -m pip install '.[dev]'
python -m pytest -q

CI runs the full suite on Python 3.10, 3.12, and 3.13.

See INTEGRATIONS.md for agent setup, CHANGELOG.md for release history, and VALIDATION.md for validation limits.