logzip 2.1.3

Compress logs for LLM analysis (Rust-powered)

logzip (Rust)

Compress logs before sending to LLM. Powered by Rust & PyO3.

raw log → [logzip compress] → compressed text → LLM (Claude Code / Cursor / API)

Before / After

Raw Log (Uvicorn):

INFO: 127.0.0.1:45678 - "GET /api/v1/status HTTP/1.1" 200 OK
INFO: 127.0.0.1:45679 - "GET /api/v1/status HTTP/1.1" 200 OK
... (100 similar lines) ...

logzip output:

--- PREFIX ---
INFO: 127.0.0.1:
--- LEGEND ---
#0# = - "GET /api/v1/status HTTP/1.1" 200 OK
--- BODY ---
45678 #0#
45679 #0#
...

Typical savings: 52–58% on structured logs (systemd, uvicorn, docker).
Anomalies and unique lines stay uncompressed — visible at a glance in the BODY.

Why use logzip? (RAG & LLM)

When working with logs in LLMs (Claude, GPT, RAG systems), you face two problems:

1. Context Limit: Logs are huge. A 10MB log is ~2.5M tokens.
2. Noise: 90% of the log consists of repeating INFO and identical requests that drown out the real error.

logzip is well-suited for RAG pipelines: it compresses the context before sending it to the model, saving money on tokens and increasing answer accuracy by highlighting anomalies.

---

Performance (7.96 MB Log, ~2M tokens)

Benchmarked on a real 7.96 MB production log.

logzip modes

Mode	CLI	Time (ms)	Size (KB)	Saved (%)	Output type
fast	--quality fast	~200	~4,900	~40%	text/LLM
balanced	--quality balanced	404	3,928	52%	text/LLM
balanced + 2 passes ★	--quality balanced --bpe-passes 2	418	3,404	58%	text/LLM
max	--quality max	507	3,511	57%	text/LLM

★ Recommended. A second compression pass finds repeated token sequences in already-compressed text — 14 ms overhead, 7% more savings vs balanced.

--quality max uses a larger legend (512 vs 128 entries) which adds overhead without a second pass benefit. Use --bpe-passes 2 with balanced instead.

vs. binary compressors (for context)

Tool	Time (ms)	Size (KB)	Saved (%)	LLM-readable?
lz4	6	1,280	84%	No
zstd (lvl 3)	14	819	90%	No
zlib (lvl 6)	69	840	90%	No
logzip (recommended)	418	3,404	58%	Yes

Binary compressors produce opaque binary blobs — LLMs cannot read them. logzip trades ~30% size for fully human- and LLM-readable output.

Token estimation: 1 token ≈ 4 characters (rough estimate for English-like logs).

Economic Impact

┌──────────────────────────────────────────────────────────┐
│  logzip Savings (7.96 MB Production Log)                 │
├──────────────────────────────────────────────────────────┤
│  Raw Size:        8,151 KB  (~1,990,000 tokens)          │
│  After balanced:  3,928 KB  (~959,000 tokens,  -52%)     │
│  After 2 passes:  3,404 KB  (~831,000 tokens,  -58%)     │
├──────────────────────────────────────────────────────────┤
│  Cost Before:     $5.97                                  │
│  Cost After:      $2.49      (Claude 3.5 Sonnet Input)   │
│  LLM Efficiency:  2.4x larger context for the same price │
└──────────────────────────────────────────────────────────┘

---

Install

Python API + logzip-py CLI:

pip install logzip

Rust CLI + MCP Server:

cargo install logzip

CLI

Two CLIs are available. Both provide compress and decompress subcommands with identical flags.

Rust binary (cargo install logzip → logzip):

# stdin → stdout
logzip compress < app.log

# quality preset (fast|balanced|max)
logzip compress --quality balanced < app.log

# recommended: balanced + second pass
logzip compress --quality balanced --bpe-passes 2 < app.log

# with preamble (LLM decode instructions at the top)
logzip compress --preamble < app.log > compressed.txt

# save + show stats
logzip compress --stats -i app.log -o app.logzip

# lossless timestamps: keep full sub-second precision (default trims to milliseconds)
logzip compress --exact-timestamps -i app.log -o app.logzip

# decompress
logzip decompress -i app.logzip

Python CLI (pip install logzip → logzip-py):

# same flags as above, plus:

# explicit profile (otherwise auto-detected)
logzip-py compress --profile journalctl < /tmp/syslog.txt

Python API

from logzip import compress, decompress

# compress
result = compress(raw_log_text)
print(result.render(with_preamble=True))   # → for LLM
print(result.stats_str())                  # → for logs

# fine-grained control
result = compress(
    raw_log_text,
    max_legend_entries=128,   # legend size
    bpe_passes=2,             # second-pass compression (compresses repeated token sequences)
    do_normalize=True,        # collapse timestamps, ANSI, IPs
    do_templates=True,        # structural template extraction
    exact_timestamps=False,   # True → keep full sub-second timestamp precision
)

# decompress
original = decompress(result.render())

MCP Server (Local — Claude Desktop / Claude Code)

Requires the Rust binary (cargo install logzip, see Install).

Add to your claude_desktop_config.json:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "logzip": {
      "command": "logzip",
      "args": ["mcp", "--allow-dir", "/var/log", "--allow-dir", "/home/user/logs"]
    }
  }
}

Or add via Claude Code CLI:

claude mcp add logzip -- logzip mcp --allow-dir /var/log

Available tools

Tool	Description
compress_content(content, quality)	Compress log text pasted directly into the conversation
get_stats(path)	File size, token estimate, detected profile — call first to decide strategy
compress_file(path, quality)	Compress entire file — for files < 200 K tokens
compress_tail(path, lines, quality)	Compress last N lines — efficient for large files

Available prompts

Prompt	Description
analyze_logs	Compresses the log server-side and prepares an SRE analysis context

How to use

LLMs don't automatically pick up MCP tools — you need to reference them explicitly. Two ways:

Option A — explicit ask (works everywhere):

"Use logzip to analyze /var/log/syslog"

Option B — analyze_logs prompt (Claude Code):

/mcp → logzip → analyze_logs → path: /var/log/syslog

This compresses the log server-side and drops an SRE-ready context into the conversation.

Option C — install the log-analysis skill (Claude Code, recommended):

The skill makes Claude automatically reach for logzip whenever you mention a log file — no explicit instruction needed.

# 1. Register this repo as a single-plugin marketplace
#    (reads .claude-plugin/marketplace.json from the repo root)
claude plugin marketplace add NailShakurov/logzip

# 2. Install the logzip plugin from that marketplace
claude plugin install logzip@logzip

The install target is <plugin-name>@<marketplace-name> — both are logzip here (the marketplace name comes from the name field in marketplace.json, not the repo path). To pull later updates, run claude plugin marketplace update logzip.

After that, asking "what's in /var/log/syslog?" is enough — Claude calls get_stats and compress_tail on its own.

Security

The MCP server only reads files inside directories specified via --allow-dir.
If no --allow-dir is given, defaults to the current working directory.
All paths are canonicalized before comparison to prevent path traversal attacks.

---

Through the eyes of an LLM

Unlike gzip/zstd which produce binary noise, logzip produces structured text. The model reads the legend once and works with the compressed body directly — it doesn't need to expand every token to understand the log.

Input for LLM:

This is a compressed log. Rules: #0# is replaced by GET /api/v1/status.

--- BODY --- 12:00:01 #0# 200 OK 12:00:02 #0# 500 ERR <-- Boom, anomaly!

The model instantly spots the 500 error without wading through thousands of identical successful requests.

Architecture & Safety

flowchart TD
    A([raw log text]) --> B

    subgraph pipe["Compression Pipeline"]
        B["① Profile Detection\njournalctl · docker · uvicorn · nodejs · plain"]
        C["② Normalizer\nANSI · timestamps · hex zeros · common prefix"]
        D["③ Frequency Analysis\nparallel n-gram counting — rayon"]
        E["④ Preserve Filter\nUUID · IPv4 · hex≥16 · custom regex\nkeeps diagnostic IDs in body"]
        F["⑤ Greedy Legend Builder\nO(N) positional index — up to 512 entries"]
        G["⑥ AhoCorasick Substitution\nsingle-pass k-way merge"]
        H{bpe_passes > 0?}
        I["⑦ Recursive BPE\n2nd-pass on compressed body"]
        J["⑧ Template Extraction\nstructural repeats → &tag = value"]

        B --> C --> D --> E --> F --> G --> H
        H -->|yes| I --> J
        H -->|no| J
    end

    J --> K([CompressResult])
    K --> L[body]
    K --> M[legend]
    K --> N[templates]
    K --> O[stats]

1. Normalizer: Collapses ANSI, timestamps, IPs, and common prefixes.
2. Frequency Analysis: Parallel n-gram counting using rayon.
3. Preserve Filter: Skips UUID, IPv4, long hex, and custom patterns — keeps them visible in the body for LLM analysis.
4. Greedy Legend: Optimized selection using a positional index (O(N)).
5. Direct Replacement: Fast substitution without re-scanning.
6. Second Pass: Compresses repeated token sequences in the already-compressed body.
7. Templates: Structural template extraction.

Safety First

• Pure Rust: Core logic is 100% Rust.
• Zero unsafe: The codebase contains no unsafe blocks, ensuring memory safety within the Python runtime.
• Stress-tested: Handled multi-GB logs without memory leaks or crashes.

Reproducibility

Want to verify our benchmarks? Run the included script:

python benchmark.py

Roadmap

Priority:

• Streaming mode for multi-GB logs

Planned:

• MCP server for Claude Code
• Suffix automaton for arbitrary repetition search