Why trimtok?

Autonomous coding agents burn 60%–85% of their context window on high-volume, low-entropy noise: 2,000-line passing pytest runs, deep framework stack frames (site-packages, node_modules), unchanged git diff context lines, repetitive grep matches, and deeply nested JSON tool responses.

Naive truncation (head -n 50 or character slicing) destroys critical failure signals hidden in the middle of logs. Summarizing via a secondary LLM adds seconds of latency and risks hallucinating line numbers or error codes.

trimtok solves this deterministically, 100% locally and offline, in ~1–2 ms:

• Stage 1 — Terminal Pipeline (compress_terminal): Strips ANSI/spinner artifacts, prunes intermediate framework stack frames while preserving entrypoints and crash sites, condenses unchanged git diff context while keeping file headers and @@ hunk scopes, folds exact and Drain-template log repeats, and turns repeated test/build commands into a compact Cross-Turn Stateful Delta.
• Stage 2 — Tool & Code Pipeline (compress_tool_result): Deduplicates repeated spans across turns, groups multi-file grep hits under enclosing def/class scopes, condenses large source files via AST Code Skeletonization (keeping imports, signatures, type annotations, and docstrings while folding function bodies), and transcodes nested JSON payloads into lossless Hierarchical TOON Tables verified by an in-memory round-trip check.
• Safety Rail — Content-Addressed Context Recovery (CCR): Every compressed payload stores its exact, unmodified original in a global SQLite WAL + Zstandard store (~/.trimtok), addressable by a short Base62 #c1 alias with instant section slicing (errors, diff, summary, tail), safe regex grep, or full restoration.

Visual Before & After Showcase

1. Cross-Turn Stateful Command Delta (delta_engine.py)

When an agent runs pytest a second time after editing code (even with shifted line numbers, updated timings, or flag changes like pytest -v vs pytest -q):

============================= test session starts =============================
platform win32 -- Python 3.12.4, pytest-8.3.2, pluggy-1.5.0
rootdir: /workspace/app
collected 201 items

tests/test_auth.py ............................                         [ 13%]
tests/test_db.py ...................................................... [ 40%]
tests/test_api.py ..................................................... [ 67%]
... (1,950 lines of passing test output & noisy INFO logs) ...
================================== FAILURES ===================================
__________________________________ test_boom __________________________________
    def test_boom():
>       assert compute_status() == 1
E       AssertionError: expected 1 got 2
=========================== short test summary info ===========================
FAILED tests/test_1000.py::test_boom - AssertionError: expected 1 got 2
======================== 1 failed, 200 passed in 43.2s ========================

[tt:delta cmd="pytest -q" sim=99% +#1 -#1 !#1 #c2 #c1]
[tt:legend] Delta vs previous run of the same command: + added, - resolved/removed, ! still failing; refs = full snapshots via trimtok_retrieve.
--- STILL FAILING / ACTIVE ERRORS (1) ---
! FAILED test_1000.py::test_boom - AssertionError: expected 1 got 2
--- ADDED / CHANGED IN THIS TURN (+1) ---
+ ===== 1 failed, 200 passed in 43.2s =====
--- RESOLVED / REMOVED (-1) ---
- ===== 1 failed, 200 passed in 42.1s =====

2. Lossless Hierarchical TOON JSON Transcoding (toon_codec.py)

Verbose JSON arrays and API responses are unnested via dot-notation, common URL/path prefixes are factored into @prefix, and every table is decoded in-memory to verify recursive strict equality (_strict_eq) before emission:

{
  "meta": {"status": "success"},
  "data": {
    "results": [
      {
        "id": 0,
        "title": "Item 0",
        "url": "https://example.com/items/0",
        "tags": ["dev", "lead"]
      },
      {
        "id": 1,
        "title": "Item 1",
        "url": "https://example.com/items/1",
        "tags": ["qa"]
      },
      {
        "id": 2,
        "title": "Item 2",
        "url": "https://example.com/items/2",
        "tags": ["pm", "ops"]
      }
    ]
  }
}

[tt:legend] TOON table (lossless): "@prefix(A)=base" = shared path shorthand; "@rows[N]{f1,f2}:" header, then N rows, cells split by "|" (quoted = literal).
@prefix(~/)=https://example.com/items/
@meta:{"status":"success"}
@data.results[3]{id,title,url,tags}:
0|Item 0|~/0|["dev","lead"]
1|Item 1|~/1|["qa"]
2|Item 2|~/2|["pm","ops"]

3. Structured Traceback Pruning & AST Code Skeletonization (pruners.py)

Deep stack traces collapse intermediate library frames (site-packages, node_modules) while keeping 100% of the entrypoint, user project frames, crash frame, and exception line. Large source files keep class/function signatures, type annotations, and docstrings while folding long implementation bodies:

Traceback (most recent call last):
  File "/app/main.py", line 25, in handle_request
    result = pipeline.process(data)
  [... 14 framework frames in site-packages/starlette & fastapi omitted ...]
  File "/app/pipeline.py", line 58, in process
    return 100 / count
ZeroDivisionError: division by zero

Quick Start

1. Universal One-Command Auto-Install (Recommended)

Install trimtok and automatically register its stdio MCP server across every supported AI agent CLI and IDE on your machine:

pip install trimtok
trimtok install

trimtok install detects and atomically updates:

• Claude Code (~/.claude.json via claude mcp add-json) & Claude Desktop (claude_desktop_config.json)
• Google Antigravity & Antigravity IDE (~/.gemini/config/mcp_config.json, ~/.gemini/antigravity/mcp_config.json)
• Gemini CLI (~/.gemini/settings.json) & Codex CLI (~/.codex/config.toml with TOML validation)
• Cursor (~/.cursor/mcp.json), Windsurf, VS Code (Copilot), OpenCode, and Cline

2. Standalone CLI Usage

# Inspect compression savings and global CCR cache health (~/.trimtok)
trimtok status

# View telemetry breakdown by command shape and retrieval hit rates
trimtok usage

# Compress any noisy log or file directly from shell pipelines
cat build.log | trimtok compress
trimtok compress --file pytest_output.txt

# Retrieve uncompressed originals or targeted sections by #cN alias
trimtok retrieve "#c1"
trimtok retrieve "#c1" --section errors
trimtok retrieve "#c1" --section diff

# Purge expired or all cached originals and vacuum SQLite store
trimtok purge

3. Hermes Agent Native Plugin

# Linux / macOS
git clone https://github.com/Stxyu-p/trimtok.git ~/.hermes/plugins/trimtok

# Windows (PowerShell)
git clone https://github.com/Stxyu-p/trimtok.git "$env:LOCALAPPDATA\hermes\plugins\trimtok"

hermes plugins enable trimtok

How the Dual-Stage Pipeline Works

flowchart TD
    In(["Raw Terminal / Tool Output"]) --> SizeGate{"Size >= min_chars<br/>(default 2 KB)?"}
    SizeGate -- "No" --> Pass["Passthrough Untouched"]
    SizeGate -- "Yes" --> Verbatim{"Verbatim Reference Tool?<br/>(e.g. read_file)"}
    Verbatim -- "Yes" --> IndexOnly["Index Spans for Future Turns<br/>& Return Verbatim"]
    Verbatim -- "No" --> StageRoute{"Pipeline Stage"}

    StageRoute -- "Terminal Output" --> S1["Stage 1: Terminal Pipeline<br/>• Strip ANSI & spinners<br/>• Prune framework tracebacks<br/>• Fold unchanged git diff context<br/>• Drain template & exact repeat fold<br/>• 2-Stage Fuzzy Command Delta"]
    StageRoute -- "Tool / JSON / Code" --> S2["Stage 2: Tool Pipeline<br/>• Scoped grep grouping by def/class<br/>• AST Code Skeletonization<br/>• Cross-turn span & 3-gram dedup<br/>• Lossless TOON JSON transcoding"]

    S1 --> Gates{"Safety & Quality Gates<br/>1. 100% Fault Lines Preserved?<br/>2. BPE Tokens Strictly Reduced?<br/>3. Within guard_ms Deadline?"}
    S2 --> Gates

    Gates -- "Any Gate Fails" --> Pass
    Gates -- "All Gates Pass" --> CCR["Store Original in ~/.trimtok<br/>(SQLite WAL + Zstandard)<br/>& Emit Compressed + #cN Ref"]

Core Engineering Guarantees

Content-Addressed Context Recovery (CCR)

Whenever trimtok compresses an output, the full unmodified original is persisted in the global CCR Store (~/.trimtok/trimtok_ccr_v2.sqlite3) using SQLite WAL mode and Zstandard payload compression.

trimtok_retrieve Modes (MCP & CLI)

Architecture & Modules

Performance & Benchmarks

Latency Benchmarks (python tests/test_benchmark.py)

Configuration & CLI Reference

MCP Tools Exposed (trimtok.mcp_server)

Recommended Ecosystem Combo

While trimtok works standalone as your context compression shield, pairing it with MemCore and Skill Proof completes the Global Agent Trinity — connecting every AI coding client on your machine (Claude Code, Cursor, Antigravity, Gemini CLI, Codex, VS Code Copilot, Windsurf, OpenCode, Cline, and Hermes) to three unified local hubs:

flowchart LR
    Task(["Developer Request"]) --> SP["Skill Proof<br/>~/.skill-proof/skills<br/>(Deterministic Skill Router)"]
    SP --> MC["MemCore<br/>~/.memcore<br/>(Governed Bitemporal Memory)"]
    MC --> Agent["Any AI Coding Agent"]
    Agent --> TT["trimtok<br/>~/.trimtok<br/>(88.3% Context Compression + CCR #c1)"]
    TT --> Agent

# Install & link the complete Global Agent Trinity across all your IDEs and CLIs
pip install trimtok memcore skill-proof
trimtok install && memcore init && memcore install && skill-proof install
skill-proof vault migrate

Verification & License

pytest -q
python tests/test_core.py
python tests/test_v3_upgrade.py
python tests/test_dedup.py
python tests/test_torture.py
python tests/test_benchmark.py
python tests/stress_heavy.py

Distributed under the MIT License. Designed and maintained by P Choke & SORA.