tokenjuice-py

Lean output compaction for terminal-heavy AI agent workflows.

A Python port of tokenjuice by Vincent Koc.

---

What is tokenjuice-py?

tokenjuice-py is a deterministic output compactor for terminal-heavy agent workflows. When AI agents run noisy commands like git status, docker build, pytest, or kubectl get pods, tokenjuice keeps the command semantics untouched, observes the output, and returns a smaller payload driven by JSON rule-based reducers — instead of dumping entire walls of terminal text into context.

Key benefits:

• Less transcript waste
• Fewer useless re-runs caused by context overflow
• Cleaner handoff between tools

Raw output stays available via --raw. Rules stay inspectable JSON rather than LLM heuristics.

---

Features

• 130 built-in JSON rules covering git, docker, kubectl, pytest, npm, terraform, and more
• Deterministic compaction — never changes command semantics, only removes low-signal noise
• Zero production dependencies — pure Python standard library (3.10+)
• CLI tools — reduce, wrap, reduce-json subcommands
• Extensible — user-level (~/.config/tokenjuice/rules) and project-level (.tokenjuice/rules) custom rules

---

Installation

pip install tokenjuice-py

Import name: tokenjuice
CLI command: tokenjuice

---

Quick Start

As a library

from tokenjuice import reduce_execution, ToolExecutionInput

# tokenjuice receives already-captured command output (stdout).
# It does NOT execute commands itself.
raw_stdout = (
    "On branch main\n"
    "Your branch is up to date with 'origin/main'.\n"
    "\n"
    "Changes not staged for commit:\n"
    '  (use "git add <file>..." to update what will be committed)\n'
    "        modified:   src/foo.py\n"
    "        modified:   src/bar.py\n"
    "\n"
    "Untracked files:\n"
    '  (use "git add <file>..." to include in what will be committed)\n'
    "        tests/test_new.py\n"
    "\n"
    "no changes added to commit (use \"git add\" and/or \"git commit -a\")\n"
)

result = reduce_execution(
    ToolExecutionInput(
        toolName="exec",
        command="git status",
        argv=["git", "status"],
        stdout=raw_stdout,
    )
)
print(result.inlineText)
# => "Changes not staged:\nM: src/foo.py\nM: src/bar.py\n\nUntracked files:\n?? tests/test_new.py"
print(result.stats.ratio)   # reduced chars / original chars
# => 0.21

# Clean working tree
result_clean = reduce_execution(
    ToolExecutionInput(
        toolName="exec",
        command="git status",
        argv=["git", "status"],
        stdout="On branch main\nnothing to commit, working tree clean\n",
    )
)
print(result_clean.inlineText)
# => "working tree clean"
print(result_clean.stats.ratio)
# => 0.34

CLI usage

Note: tokenjuice reduce reads from stdin or a file and uses --command only to select the matching reduction rule — it does not execute the command. Pipe the real output in:

# Pipe existing output through the reducer
git status | tokenjuice reduce --command "git status"

# One-step: wrap executes the command and compacts output automatically
tokenjuice wrap -- git status
tokenjuice wrap -- docker ps
tokenjuice wrap -- pytest tests/

# Read from a file
tokenjuice reduce build.log --command "make build"

# Output as JSON (includes classification and stats)
git status | tokenjuice reduce --command "git status" --format json

# Specify exit code for failure-aware rules
tokenjuice reduce --command "make build" --exit-code 1 < build.log

# reduce-json protocol (for AI client hooks)
echo '{"input": {"toolName": "exec", "argv": ["git", "status"], "stdout": "..."}}' \
  | tokenjuice reduce-json

# List all built-in rules
tokenjuice discover

# Verify all rules
tokenjuice verify

---

Rule System

Rule fields

Field	Type	Description
id	string	Unique identifier (e.g. git/status)
family	string	Rule family (e.g. git-status)
match	object	Match conditions (argv0, gitSubcommands, etc.)
filters	object	skipPatterns / keepPatterns
transforms	object	stripAnsi, dedupeAdjacent, etc.
summarize	object	head / tail line counts
counters	array	Count facts (e.g. number of modified files)
failure	object	Special handling on non-zero exit code

Custom rules

• User-level: ~/.config/tokenjuice/rules/
• Project-level: .tokenjuice/rules/ (project root)

Priority: project > user > builtin (later layers override by rule id).

Built-in rules (130 rules)

Category	Rule IDs
git	git/status git/diff git/diff-stat git/diff-name-only git/log-oneline git/branch git/show git/stash-list git/remote-v git/worktree-list
build	build/cmake build/dotnet build/esbuild build/go-build build/gradle build/maven build/msbuild build/pnpm-build build/swift-build build/tsc build/tsdown build/vite build/webpack build/xcodebuild
devops	devops/docker-build devops/docker-compose devops/docker-images devops/docker-logs devops/docker-ps devops/helm devops/kubectl-describe devops/kubectl-get devops/kubectl-logs devops/pulumi devops/terraform devops/terragrunt
tests	tests/jest tests/vitest tests/pytest tests/cargo-test tests/go-test tests/playwright tests/mocha tests/rspec tests/bun-test tests/swift-test tests/npm-test tests/pnpm-test tests/yarn-test
cloud	cloud/aws cloud/az cloud/gcloud cloud/gh cloud/flyctl cloud/vercel
network	network/curl network/wget network/ping network/ssh network/dig network/nslookup network/traceroute
install	install/npm-install install/npm-ci install/pnpm-install install/yarn-install install/bun-install
package	package/apt-install package/apt-upgrade package/brew-install package/brew-upgrade package/yum-install package/dnf-install package/composer package/fnm package/npm-ls
database	database/psql database/mysql database/sqlite3 database/redis-cli database/mongosh
filesystem	filesystem/ls filesystem/find filesystem/fd filesystem/rg-files filesystem/git-ls-files
service	service/systemctl-status service/journalctl service/pm2 service/lsof service/netstat service/ss service/service service/launchctl
observability	observability/top observability/htop observability/free observability/iostat observability/vmstat
search	search/rg search/grep search/git-grep
lint	lint/eslint lint/biome lint/oxlint lint/prettier-check
task	task/make task/just task/uv task/node task/python task/ruby task/php task/mise task/env
system	system/df system/du system/ps system/file
transfer	transfer/rsync transfer/scp
archive	archive/tar archive/zip archive/unzip
media	media/ffmpeg media/mediainfo
text	text/wc
generic	generic/fallback (catch-all) generic/help

Run tokenjuice discover to list all rules loaded at runtime.

---

Acknowledgements

This project is a Python port of tokenjuice, the original TypeScript implementation by Vincent Koc.

The built-in JSON rule set, core compaction concepts, and CLI design are derived from the original project. All credit for the original design goes to Vincent Koc and the tokenjuice contributors.

If you use Node.js / TypeScript or need host integrations (Cursor, Claude Code, Codex, etc.), please use the original project: https://github.com/vincentkoc/tokenjuice

---

License

MIT — see LICENSE for details.

This project includes work derived from tokenjuice, Copyright (c) 2026 Vincent Koc, also under the MIT License.