tokcodec

A codec for LLM context.
Video codecs throw away what the eye can't see. tokcodec throws away what the model doesn't need.
A 21,000-token test run becomes 342 tokens. The failure is still there.

_{Built and maintained by LNX Inc. · free of charge for everyone, forever · MIT licensed}

The problem is headroom, not just money

Every test run, install log, and 2,000-line file your AI coding agent reads lands in its context window. Most of it is filler: 400 lines of PASSED, JSON that is half whitespace, function bodies it never needed. The window fills, /compact fires, and the thread loses the plot mid-task. If you pay per token it also costs money, but on a flat plan the real price is the context you no longer have.

tokcodec makes those tool results small before they land, in graded levels, and always tells the model what it left out.

Try it in 10 seconds

npx tokcodec why path/to/any/file.py     # see the token math for yourself
npx tokcodec install                      # wire it into Claude Code (skill + hooks)

No Python on the machine? Fine. The launcher uses uvx or pipx if you have them, and otherwise downloads a private copy of uv once (about 15 MB), which brings its own Python. Prefer Python tooling directly? uvx tokcodec ... or pipx run tokcodec ... do the same. Permanent install: uv tool install tokcodec / pipx install tokcodec.

Claude Code users can also install it as a plugin, which needs no settings merge:

/plugin marketplace add lnxinc/tokcodec
/plugin install tokcodec@lnxinc

Why not just gzip it?

Because bytes and tokens are different currencies. A BPE tokenizer turns high-entropy text into roughly one token per one or two characters, and the model can't inflate gzip in its head anyway. Every byte-level trick makes the token count worse:

Token compression has to remove information the reader doesn't need, not bytes. That is what tokcodec does, in graded levels.

Benchmark

Real files, reproducible by anyone with uv run python bench/run.py (proxy tokenizer, no API key needed). Details in bench/RESULTS.md. The same table counted with Anthropic's count_tokens is in bench/RESULTS-exact.md: Claude's tokenizer counts about 1.5× more tokens on code, and the savings percentages agree within three points (−45% / −80% exact vs −48% / −83% proxy at levels 2 and 3).

Fidelity: does the model still get it right?

Token savings alone is a party trick. The question is whether a model reading the compressed version still answers correctly. bench/fidelity/ holds 104 questions with checkable answers, 8 per sample, deliberately including questions that levels 2 and 3 are expected to fail (comment contents, logic inside a collapsed body, the 10th item of a capped array). Each question is declared up front with the highest level it should survive, then asked at every level and graded. Results, Claude Sonnet 5, full report with every wrong answer unedited in bench/FIDELITY.md:

Read the bottom rows as the point, not a caveat: level 3 keeps everything it promises to keep and loses almost everything it promises to lose. One question's expected answer was corrected after the first run because the grader rejected a correct response that quoted the log line verbatim; the note is in the question file.

Run it yourself with an API key:

uv run --extra bench python bench/fidelity/run.py --estimate   # cost first, no calls
uv run --extra bench python bench/fidelity/run.py              # about $6 on Sonnet 5, cached after

What it never removes

The safety contract, in plain terms:

• Level 1 is lossless in meaning. Whitespace, ANSI codes, JSON layout, and exact duplicate lines only. Safe for anything, including edits.
• Every cut is marked. [×80], ... # 42 lines, { /* … 12 lines */ }, … [300 lines omitted]. The model always knows where to look deeper.
• Failures survive. The log codec keeps the head, the tail, and every line matching error, fail, exception, traceback, warning, or summary patterns, with a line of context.
• Outlines keep the map. Imports, classes, every signature, the first docstring line. Python outlines are valid Python; JSON stays valid JSON.
• Edits read raw. Levels 2 and 3 are for reading. The bundled skill tells Claude to read losslessly before editing, so old_string matches never fail on compressed text.
• Hooks fail open. If tokcodec is missing or crashes, the original output and exit code are shown unchanged. TOKCODEC_HOOK_DISABLE=1 turns everything off instantly.
• No telemetry, no network calls, except the optional --exact token count you ask for.

Levels

Supported languages

Outlines (level 3) for 14 languages, comment stripping for 21, plus JSON, log, diff and prose codecs. tokcodec langs prints the table. Detection is by extension or filename first, then by content for pasted snippets.

Usage

tokcodec FILE                    # level 2, kind auto-detected
tokcodec FILE -l 3 -s            # heavy level, token stats on stderr
cat big.json | tokcodec - -k json -l 1
tokcodec bench src/              # savings table for a whole directory
tokcodec count FILE              # token count
tokcodec why FILE                # the gzip table above, for any file
tokcodec langs                   # supported kinds
tokcodec install [--project] [--skill-only] [--uninstall] [--dry-run]
tokcodec serve                   # MCP server over stdio (needs tokcodec[mcp])

Add --exact to count with Anthropic's count_tokens endpoint instead of the local proxy tokenizer. It is free and needs ANTHROPIC_API_KEY or an ant auth login profile.

As a library

from tokcodec import encode

r = encode(open("app.log").read(), level=3, kind="log")
print(r.encoded)
print(r.tokens_before, r.tokens_after, r.steps)
# 15089 231 ['lossless-text', 'log-fuzzy', 'log-truncate']   (proxy counts; --exact gives Claude's)

As an MCP server

Any MCP client (Claude Code, Cursor, Zed, Claude Desktop) can call tokcodec directly:

uv tool install "tokcodec[mcp]"
claude mcp add tokcodec -- tokcodec serve

Tools: read_compact(path, level, kind), compact_text(text, kind, level), token_count(text). Each result starts with a one-line header giving the kind and before/after token counts.

Claude Code integration

Two ways in. The plugin (above) is the simplest. tokcodec install does the same by hand under ~/.claude/ (or ./.claude/ with --project). Either way you get:

1. A skill that teaches Claude when to reach for tokcodec instead of reading a huge file whole, and which level to use.
2. A Bash hook (PreToolUse) that rewrites noisy commands (pytest, npm install, cargo build, git pull, …) so their output is captured, compressed with tokcodec -k log -l 3, and handed to Claude. Claude never sees the 900 lines of dots.
3. A Read hook (PreToolUse) that stops whole-file reads above 64 KB and points Claude at tokcodec file -l 3 for an outline, then ranged reads.

The hooks are registered as tokcodec hook bash and tokcodec hook read, nothing else: no interpreter path, no home-directory path, so the same configuration works on macOS, Linux and Windows. tokcodec install merges settings, never overwrites, and --uninstall removes exactly what was added. Knobs: TOKCODEC_HOOK_LEVEL=2|3, TOKCODEC_READ_MAX_KB=64, TOKCODEC_HOOK_DISABLE=1.

Using Cursor, Codex, Aider or Gemini CLI? tokcodec is a plain stdin/stdout filter. Drop the text of SKILL.md into your agent's instructions file and pipe away, or point the agent at the MCP server.

How it works

Three things to know:

1. It's a filter, not an editor. tokcodec reads a file or a command's output and writes a shorter version to stdout. Nothing on disk changes. The compressed text exists only on its way into the model's context.
2. It shrinks tool results, not your conversation. In Claude Code, every command output and every file read lands in the context window. tokcodec makes those results smaller before they land. Your prompts and Claude's replies are untouched.
3. Lossy levels are for reading. Level 1 is safe for anything. Levels 2 and 3 change whitespace and drop detail, so an agent about to edit a file should read it losslessly. The bundled skill tells Claude exactly that.

Under the hood:

• Python goes through ast and tokenize, so a # inside a string is never mistaken for a comment and skeletons always parse.
• Brace languages (JS/TS, Go, Rust, Java, Kotlin, C#, C/C++, Swift, Dart, Scala, PHP, Zig) share one scanner that tracks strings, template literals and comments while matching braces. Control statements and type declarations are never collapsed, only function bodies.
• Ruby outlines def … end blocks by indentation.
• Logs dedupe on a fuzzy key (digits and durations normalised), then keep head, tail and every line matching error/fail/warn/exception/traceback/summary patterns, with one line of context.
• JSON is parsed and re-emitted, so output is always valid JSON.

Honest limits

• The default counter is tiktoken's o200k_base, because Anthropic doesn't publish Claude's tokenizer. Measured against count_tokens, the proxy undercounts Claude tokens by about a third on code while the savings percentages agree within three points. Headline numbers in this README are exact; --exact gives you the same.
• tokcodec does not replace prompt caching. Caching makes re-sent context cheap; tokcodec makes it smaller to begin with. Use both.
• Outlines for brace languages come from a scanner, not a parser. It handles strings, comments and nested braces, but exotic syntax (Rust raw strings with #, C# verbatim strings with "") can occasionally confuse it. When it can't find a matching brace it leaves the code untouched rather than guessing.
• Level 3 loses things on purpose. The fidelity benchmark exists to show exactly which things.

Roadmap

• Tree-sitter backend for exact outlines where the scanner falls short
• Diff-aware mode: skeleton everything except the hunks that changed
• Session-level dedupe: never send the same file twice in one conversation
• Real-task benchmark: SWE-bench-lite with and without the hooks, pass rate and total tokens
• Learned importance scoring for log lines

Contributing

git clone https://github.com/lnxinc/tokcodec && cd tokcodec
uv sync --all-extras && make test && make bench && make smoke

Every transform ships with a test that proves what it keeps: a failure line, a signature, valid JSON. See CONTRIBUTING.md and RELEASING.md. Ideas and benchmark results from your own repos are very welcome in issues.

About

tokcodec is provided by LNX Inc. free of charge for everyone. No paid tier, no telemetry, no sign-up. If it saves you tokens, a star on the repo and a link back are all we ask.

License

MIT © LNX Inc.