Tok

Tok is a local Claude Code bridge for deterministic context compression. It sits between Claude Code and the upstream model API, reduces repeated file/search/tool context when it can do so safely, and fails open to normal uncompressed behavior when fidelity is at risk.

Tok 0.2.2 is deliberately narrow: Claude Code routed through a local bridge with a local resolver beta, plus experimental local session-receipt and handoff artifacts for auditable state transfer. It is not a hosted service, agent framework, repo indexer, or general prompt-compression SDK.

Why Tok Exists

Long-running coding-agent sessions often resend verbose transcripts, file reads, search results, and tool outputs on every turn. That is useful when a human reads the output, but wasteful when the next reader is another model.

Tok tests a smaller runtime shape: compact, deterministic, model-facing state at the machine-to-machine boundary, with human-facing output preserved at the edges. Compression is rule-based rather than LLM-summarized, so behavior is repeatable and auditable.

What Tok Does

• Semantic deduplication: repeated file reads, search results, and tool outputs can be cached and replaced with compact references.
• Delta compression: changed content can be represented as a diff instead of a full repeated payload.
• Bounded rolling state: recent context stays available without unbounded history growth.
• Fail-open safety: when compression would risk fidelity, Tok serves the request in baseline mode and reports the fallback.
• Diagnostics: status, doctor, stats, logs, and optional trace audit explain what happened.

Savings

Savings are workload-dependent. Tok tends to help most on sustained sessions with repeated file reads, repeated searches, large tool outputs, or long-running debugging loops. Very short sessions may intentionally run near baseline because compression overhead is not worth paying.

Use these as practical expectations:

• Sustained sessions: meaningful input-token savings when context repeats.
• Short sessions: little or no visible savings; Tok may stay baseline.
• Risky compression cases: fallback is preferred over corrupting context.

Pricing estimates depend on provider/model rates. See docs/pricing_verification.md and docs/claims_matrix.md for the current evidence trail.

Quickstart

pip install tok-protocol
tok claude

Then, from another shell or after the session:

tok bridge status         # check bridge health
tok doctor                # explain current session state
tok stats                 # view savings
tok bridge stop           # stop cleanly

If you want an isolated CLI install and already use pipx, this works too:

pipx install tok-protocol
tok claude

tok claude starts the bridge if needed, routes Claude Code through it, and leaves your shell rc files untouched. If you prefer legacy auto-routing, opt in explicitly:

tok install --wrap-claude
source ~/.zshrc           # or source ~/.bashrc
claude

What Success Looks Like

A healthy bridge session usually has:

• tok bridge status showing the bridge running and Tok active
• tok doctor ending with Recommendation: keep Tok on
• tok stats showing With Tok vs without Tok, saved tokens, estimated savings, and net saved tokens when reacquisition overhead is present
• Degraded to baseline set to no

Representative output:

Bridge running on :9090 (PID 12345)
Saved $0.0123 - 48.1% saved
Verdict                Tok active and helping
Tok active             yes
Degraded to baseline   no
Fallbacks              0

If Degraded to baseline: yes or fallback counts rise, Tok protected the session by serving requests without compression.

Supported Surface

The public 0.2.2 workflow is:

tok init
tok install
tok claude
tok bridge status
tok doctor
tok stats
tok audit --latest
tok bridge stop
tok resolver init
tok resolver put <file>
tok resolver get tok-resolver://sha256:...

Experimental substrate commands are hidden from the default help surface:

tok session-receipt --latest --json
tok audit --session-receipt ./receipt.json --json
tok handoff export --latest --json
tok handoff inspect ./handoff.json

Session receipts summarize local bridge receipts, savings events, diagnostics, and exactness labels. Handoff packets reference a session receipt and list exact reacquisitions required before another agent performs edit-like work. These are local draft artifacts, not universal protocol or remote agent-exchange support.

The default request policy is natural-first, which preserves raw provider-compatible traffic when a turn has not yet earned compression and applies Tok only when the bridge has enough exact evidence to do so safely. For comparison or debugging, you can run without compression:

TOK_MODE=baseline tok bridge start
ANTHROPIC_BASE_URL=http://localhost:9090 claude
tok stats

For advanced routing or compatibility checks, you can still run the bridge and route a client explicitly:

tok bridge start
ANTHROPIC_BASE_URL=http://localhost:9090 <your-client-command>

That path is useful for debugging and experiments, but the low-friction public install story is pip install tok-protocol followed by tok claude.

Experimental Python submodule APIs and internal compression features exist, but they are not part of the supported 0.2.2 contract and may change without compatibility guarantees.

Streaming Behavior

Tok preserves streaming response shape for Claude Code, but the normal 0.2.x bridge path buffers the upstream stream before re-emitting it. Long responses can therefore show more first-token latency than a direct Claude Code session, followed by a quick replay of the response. Modes that prioritize correctness, including smoothness/lossless handling and extended-thinking requests, may also use the non-streaming upstream path.

Very large responses use memory proportional to response size while buffering. The default buffer limit is 100 MB and can be overridden with the TOK_MAX_STREAM_BUFFER_BYTES environment variable.

Trace Audit

Enable trace sidecars only when you want to inspect what Tok did:

TOK_TRACE=1 TOK_TRACE_CAPTURE_ARTIFACTS=1 tok bridge start
tok claude
tok audit --latest

Trace mode is local. Tok does not send trace files to the model provider, and the live trace path does not store raw prompts, responses, or tool outputs. Live session_id values are trace-local ids, not durable client ids across bridge restarts.

tok audit is useful for checking bridge behavior and exactness metadata. It is not a general protocol compliance certificate.

Prerequisites

• Python 3.10 or newer
• macOS or Linux
• Claude Code installed and available as claude
• Claude Code already configured with provider credentials

Tok is a local proxy. It does not manage API keys. If Claude Code works without Tok, tok claude should work too.

Troubleshooting

Install Verification

Use this for a clean package sanity check:

python -m venv .venv
source .venv/bin/activate
pip install tok-protocol
tok --version
tok --help
tok claude --help
tok install
tok bridge start --help
tok bridge status --help
tok stats --help
tok audit --help

For local checkout work:

pip install .
tok --version
tok --help

Docs Map

Start here:

• docs/agent_quickstart.md: cold-clone workflow for autonomous agents
• docs/repository_map.md: directory map and public/internal boundaries
• docs/bridge.md: full bridge tutorial
• docs/cli-reference.md: supported CLI surface
• docs/troubleshooting.md: fallback, logs, degraded sessions, savings interpretation
• docs/diagnostics.md: detailed bridge health signals

For release and architecture context:

• CHANGELOG.md: release notes
• docs/public-release-decision.md: supported workflows, limitations, and release bar
• docs/spec/README.md: Tok Trace draft specification map
• docs/architecture.md: current architecture
• docs/architecture-0.2.md: roadmap, not the current runtime contract
• docs/production-readiness.md: advanced release posture

Repo Map

• src/tok/: runtime, bridge, CLI, and library code
• docs/: public product docs plus release/reference docs
• docs/spec/: draft Tok Trace and protocol-layer specification work
• docs/maintainers/: maintainer roadmap and planning notes
• docs/plans/: planning artifacts outside the default onboarding path
• ops/: internal tracking ledgers, not public product docs
• examples/: experimental wrapper/API examples outside the default bridge-first path
• tests/: unit, integration, replay, smoke, and stability coverage

For the fuller directory map, see docs/repository_map.md.

Development

For maintainer validation:

uv sync --frozen --extra dev
uv run python scripts/run_agent_smoke.py
uv run pre-commit run --all-files
uv run ruff check src/tok tests
uv run mypy src/tok
uv run pytest tests/unit tests/integration -v --cov=src/tok --cov-fail-under=80
uv build

For release-specific checks, see docs/release-checklist.md and docs/CICD_INTEGRATION.md.

Agent Operation

Coding agents should start with AGENTS.md and docs/agent_quickstart.md. They define the supported 0.2.0 surface, verification commands, live-bridge reporting rules, extension boundaries, and claims agents must not make without evidence.

Privacy

Tok runs locally. No data leaves your machine except the model/API calls Claude Code would already make. Optional trace sidecars are local metadata files and do not store raw prompts, responses, or tool outputs by default.

Support Tok

Tok exists because repeated machine-facing context is a real cost in long coding-agent sessions. The most useful support is practical feedback:

• Star the repo and share it with developers who use Claude Code heavily.
• File issues with bridge logs, tok doctor, and tok stats output.
• Share benchmark results from real sustained sessions.
• Contribute docs, tests, or focused fixes.