What is Claw Compactor?

Claw Compactor is an open-source LLM token compression engine built around a 14-stage Fusion Pipeline. Each stage is a specialized compressor — from AST-aware code analysis to JSON statistical sampling to simhash-based deduplication — chained through an immutable data flow architecture where each stage's output feeds the next.

Demo

$ claw-compactor benchmark ./my-workspace

  Claw Compactor v7.0 — Fusion Pipeline Benchmark
  ─────────────────────────────────────────────────

  Scanning workspace... 47 files, 234,891 tokens

  Stage Results:
  ┌──────────────────┬──────────┬───────────┬──────────┐
  │ Stage            │ Applied  │ Reduction │ Time     │
  ├──────────────────┼──────────┼───────────┼──────────┤
  │ Cortex           │ 47/47    │ —         │ 12ms     │
  │ Photon           │ 3/47     │ 2.1%      │ 4ms      │
  │ RLE              │ 41/47    │ 8.3%      │ 6ms      │
  │ SemanticDedup    │ 47/47    │ 12.7%     │ 18ms     │
  │ Ionizer          │ 8/47     │ 71.2%     │ 9ms      │
  │ Neurosyntax      │ 23/47    │ 18.4%     │ 31ms     │
  │ TokenOpt         │ 47/47    │ 4.1%      │ 3ms      │
  │ Abbrev           │ 12/47    │ 6.8%      │ 5ms      │
  └──────────────────┴──────────┴───────────┴──────────┘

  Summary:
    Before:  234,891 tokens ($2.35 at GPT-4 rates)
    After:   108,250 tokens ($1.08)
    Saved:   126,641 tokens (53.9%) — $1.27/run
    Time:    88ms total

  Estimated monthly savings at 100 runs/day: $3,810

How It Compares

Why Claw Compactor wins: LLMLingua-2 drops tokens by perplexity score — effective for natural language, but destroys code identifiers, JSON keys, and log patterns. Claw Compactor uses content-type-aware stages that understand the structure of what they're compressing.

Input
  |
  v
┌─────────────────────────────────────────────────────────────────────────┐
│                         FUSION PIPELINE                                 │
│                                                                         │
│  QuantumLock ─> Cortex ─> Photon ─> RLE ─> SemanticDedup ─> Ionizer    │
│       |            |         |        |          |              |        │
│   KV-cache    auto-detect  base64   path     simhash       JSON         │
│   alignment   16 languages  strip  shorten   dedup        sampling      │
│                                                                         │
│  ─> LogCrunch ─> SearchCrunch ─> DiffCrunch ─> StructuralCollapse      │
│        |              |              |                |                  │
│    log folding    result dedup   context fold    import merge            │
│                                                                         │
│  ─> Neurosyntax ─> Nexus ─> TokenOpt ─> Abbrev ─────────> Output       │
│        |             |          |           |                            │
│    AST compress   ML token   format     NL shorten                      │
│    (tree-sitter)  classify   optimize   (text only)                     │
│                                                                         │
│  [ RewindStore ] ── hash-addressed LRU for reversible retrieval         │
└─────────────────────────────────────────────────────────────────────────┘

Key design principles:

• Immutable data flow — FusionContext is a frozen dataclass. Every stage produces a new FusionResult; nothing is mutated in-place.
• Gate-before-compress — Each stage has should_apply() that inspects context type, language, and role before doing any work. Stages that don't apply are skipped at zero cost.
• Content-aware routing — Cortex auto-detects content type (code, JSON, logs, diffs, search results) and language (Python, Go, Rust, TypeScript, etc.), then downstream stages make type-aware compression decisions.
• Reversible compression — Ionizer stores originals in a hash-addressed RewindStore. The LLM can call a tool to retrieve any compressed section by its marker ID.

Benchmarks

Real-World Compression (FusionEngine v7 vs Legacy Regex)

SWE-bench Real Tasks

Tested on real SWE-bench instances with actual repository code:

vs LLMLingua-2 (ROUGE-L Fidelity)

Claw Compactor preserves more semantic content at the same compression ratio, with zero LLM inference cost.

Quick Start

Install from PyPI

pip install claw-compactor

Or clone from source

git clone https://github.com/open-compress/claw-compactor.git
cd claw-compactor
pip install -e .

Run

# Benchmark your workspace (non-destructive)
claw-compactor benchmark /path/to/workspace

# Full compression pipeline
claw-compactor compress /path/to/workspace

Requirements: Python 3.9+. Optional: pip install claw-compactor[accurate] for exact token counts via tiktoken.

API

FusionEngine — Single Text

from scripts.lib.fusion.engine import FusionEngine

engine = FusionEngine()

result = engine.compress(
    text="def hello():\n    # greeting function\n    print('hello')",
    content_type="code",    # or let Cortex auto-detect
    language="python",      # optional hint
)

print(result["compressed"])     # compressed output
print(result["stats"])          # per-stage timing + token counts
print(result["markers"])        # Rewind markers for reversibility

FusionEngine — Chat Messages

messages = [
    {"role": "system", "content": "You are a coding assistant..."},
    {"role": "user", "content": "Fix the auth bug in login.py"},
    {"role": "assistant", "content": "I found the issue. Here's the fix:\n```python\n..."},
    {"role": "tool", "content": '{"results": [{"file": "login.py", ...}, ...]}'},
]

result = engine.compress_messages(messages)

# Cross-message dedup runs first, then per-message pipeline
print(result["stats"]["reduction_pct"])   # aggregate compression %
print(result["per_message"])              # per-message breakdown

Rewind — Reversible Retrieval

engine = FusionEngine(enable_rewind=True)
result = engine.compress(large_json, content_type="json")

# LLM sees compressed output with markers like [rewind:abc123...]
# When the LLM needs the original, it calls the Rewind tool:
original = engine.rewind_store.retrieve("abc123def456...")

Custom Stage

from scripts.lib.fusion.base import FusionStage, FusionContext, FusionResult

class MyStage(FusionStage):
    name = "my_compressor"
    order = 22  # runs between StructuralCollapse (20) and Neurosyntax (25)

    def should_apply(self, ctx: FusionContext) -> bool:
        return ctx.content_type == "log"

    def apply(self, ctx: FusionContext) -> FusionResult:
        compressed = my_compression_logic(ctx.content)
        return FusionResult(
            content=compressed,
            original_tokens=estimate_tokens(ctx.content),
            compressed_tokens=estimate_tokens(compressed),
        )

# Add to pipeline
pipeline = engine.pipeline.add(MyStage())

The 14 Stages

Each stage is independent and stateless. Stages communicate only through the immutable FusionContext that flows forward through the pipeline.

Workspace Commands

python3 scripts/mem_compress.py <workspace> <command> [options]

Options: --json, --dry-run, --since YYYY-MM-DD, --quiet

Architecture

See ARCHITECTURE.md for the full technical deep-dive:

• Immutable data flow design
• Stage execution model and gating
• Rewind reversible compression protocol
• Cross-message semantic deduplication
• How to extend the pipeline

12,000+ lines Python  ·  1,600+ tests  ·  14 fusion stages  ·  0 external ML dependencies

Installation

# Clone
git clone https://github.com/open-compress/claw-compactor.git
cd claw-compactor

# Optional: exact token counting
pip install tiktoken

# Optional: AST-aware code compression (Neurosyntax)
pip install tree-sitter-language-pack

# Development
pip install -e ".[dev,accurate]"

Zero required dependencies. tiktoken and tree-sitter are optional enhancements — the pipeline runs with built-in heuristic fallbacks for both.

Who Uses This

Using Claw Compactor? Open a PR to add yourself here.

Project Stats

Contributing

See CONTRIBUTING.md for guidelines on:

• Setting up the development environment
• Adding new Fusion stages
• Running the test suite
• Submitting PRs

Related

• OpenClaw — AI agent platform
• ClawhubAI — Agent skills marketplace
• OpenClaw Discord — Community
• OpenClaw Docs — Documentation
• Full Documentation — GitHub Pages docs

token-compression llm-tools fusion-pipeline reversible-compression ast-code-analysis context-compression ai-agent openclaw python developer-tools

License

MIT