Skip to main content

log-ai-compressor

Every finding carries line numbers + the sha256 of the cited text. Change the cited text — line endings normalized — and verify returns TAMPERED.

For ops / backend / security engineers reading logs. Analysis runs locally — 0 API calls, logs stay on your machine. The optional AI interpretation layer is off by default; if you enable it, only redacted excerpts are sent to the provider you configure, with preview and explicit consent before sending.

verify re-reads the raw log and checks every quote against the cited text. If the cited text was edited, it exits 1 — you never have to trust the tool. Line endings (CR/LF) are normalized per the stated contract and do not count as tampering.

Metric scope: PA 85.05% uses Loghub Content (timestamps/hosts/levels stripped, Drain protocol); the same implementation on raw input scores 31.08%. Same PA metric and annotated-line denominator, evaluated under different input protocols (Loghub Content vs raw); do not mix the two results.

$ log-ai-compressor verify --report examples/app_report.json
模式:复核已导出报告
对象:报告 app_report.json
证据:共 4 条引用,匹配 3 条,失配 1 条,源不可读 0 条,缺行号 0 条,缺摘要 0 条

自校验结论:TAMPERED        # 退出码 1

  ✗ MISMATCH c3801f905159 examples\app.log 期望 bc5b93cec7cde01c 实际 1b8242c1671f1e29

Key lines from the run (Chinese CLI; stdout + stderr, so the MISMATCH line may appear before or after the verdict): 4 quotes checked, 3 matched, 1 mismatched → verdict TAMPERED, exit 1. The full run also ends with a one-line explanation. Hashes and exit codes are byte-identical to what you will get; reproduce it with the cited-text edit below.

▶ Install & self-proof below · 中文 README

$ python -m pip install -i https://pypi.org/simple "log-ai-compressor==3.0.5"
$ python -c "import log_ai_compressor, pathlib, shutil, os; os.makedirs('examples', exist_ok=True); shutil.copy(pathlib.Path(log_ai_compressor.__file__).parent/'data/examples/app.log', 'examples/app.log')"
$ log-ai-compressor run examples/app.log -f json -o examples/app_report.json
$ log-ai-compressor verify --report examples/app_report.json
自校验结论:VERIFIED

The last line is verbatim output (Chinese CLI): verdict VERIFIED, all 4 quotes matched, exit 0. No clone needed: line 2 copies the sample log out of the installed wheel. The key lines at the top are from a real run too: change the cited db-primary to db-primarY in examples/app.log, re-run verify, and you get exactly that. Change it back → VERIFIED again. There is no "mostly passed" tier.

Release Python License

Repository is hosted on Gitee (reachable worldwide, but slow from some regions). The package itself installs from PyPI with no dependency on Gitee. Screenshots live in the Chinese README — this page stays text-only so every line renders anywhere.

Open the UI (the recommended entry point)

Your machine One command
Windows double-click start.bat in the repo root — finds Python, installs what is missing, opens the browser
anything else pip install "log-ai-compressor[web]==3.0.5" && log-ai-compressor web

Three steps to a conclusion: ① drag a log into the window (or pick a local file with Browse) → ② levels default to ERROR/FAIL/WARN only → ③ press Start. The first thing you get is one card — located root cause, or "insufficient evidence" with the exact gap — not a wall of template clusters. Press Re-check evidence and the tool re-reads the raw log line by line and returns VERIFIED / TAMPERED.

The UI is zero-build and zero-CDN, so it works offline; the file browser picks local paths and never uploads. The one exception is FastAPI's own /api/docs (Swagger), which pulls its stylesheet from jsdelivr — the main screen is unaffected.

CLI (no UI: pipes, CI, agents)

The install and sample commands are in the first screen above. After them:

log-ai-compressor run examples/app.log -f summary

Pin the version: mirrors lag by days, and 2.13.1 (an older release) has no -f summary. Samples ship inside the wheel (log_ai_compressor/data/examples/); if examples/ is absent, run / compare fall back by basename and print a notice. Verbatim output of the command above (the CLI prints Chinese — numbers, fingerprint and hashes are byte-identical to what you will get):

【日志分析摘要】examples\app.log
原始日志 ≈379 tokens → 摘要 ≈234 tokens(压缩 38%)
确定性指纹 `48e5781aafc4b27fe0f68c5847afe6d891936ac9f2e0af59aefaa9860f9c2886`(同一份日志跑两遍必须完全一致)
总行数 24,错误 8 行,去重后 4 种,时间范围 2024-03-15 08:00:00 ~ 2024-03-15 08:00:25。
已定位根因:java.net.ConnectException: Connection refused(另有 1 个根因候选)

(证据出处末尾 16 位十六进制 = 该段原文 sha256 前 16 位,可自行复核)
1. [P0 | ERROR | ×3 | 根因 | 行 4~13] authorization failed: connection refused to db-primary:5432
   证据出处:L4 bc5b93cec7cde01c
   -> 因果链源头(1 个错误由其衍生)
2. [P0 | ERROR | ×3 | 行 6~11] request 8812 failed: upstream unavailable
   证据出处:L6 21d12907c3115d34
3. [P0 | ERROR | ×1 | 根因 | 行 18~22] java.net.ConnectException: Connection refused
   证据出处:L18-L22 68af24f230baff5c
   -> Caused-by 因果链直连,根因:java.net.ConnectException: Connection refused
4. [P0 | ERROR | ×1 | 行 23~23] stock check timeout after 3000ms item=SKU-7781
   证据出处:L23 a72b9f38845a14ed

Prove it to yourself: edit the cited text

log-ai-compressor run examples/app.log -f json -o examples/app_report.json
log-ai-compressor verify --report examples/app_report.json   # VERIFIED, exit 0

Now change db-primary to db-primarY in examples/app.log and re-run the second command. Key lines (Chinese CLI; the MISMATCH line is stderr, so its order relative to the verdict can vary; the full run ends with a one-line explanation):

模式:复核已导出报告
对象:报告 app_report.json
证据:共 4 条引用,匹配 3 条,失配 1 条,源不可读 0 条,缺行号 0 条,缺摘要 0 条

自校验结论:TAMPERED

  ✗ MISMATCH c3801f905159 examples\app.log 期望 bc5b93cec7cde01c 实际 1b8242c1671f1e29

Translation: 4 quotes checked, 3 matched, 1 mismatched → verdict TAMPERED, exit 1. Change it back → VERIFIED again. No "mostly passed" tier. Machine-readable: add --json and read checked: 4 / matched: 3 / mismatched: 1.

What it is (and what it is not)

grep / tail Drain3 3-shot LLM this tool
What you get matching lines templates templates conclusion + line range + sha256
Model needed no no yes no (core analysis: 0 API calls)
Full 10-set GA — 71.58% not run 88.68%
Full 10-set PA — 58.06% not run 85.05%
Output you can re-check yes no no yes (verify re-reads the log)

Do the numbers hold up?

Full Loghub 2.0, ten datasets: 1,940,760 lines / 1,621 clusters.

  • GA (grouping accuracy): 88.68% vs Drain3 71.58% — same tokenization, same scoring
  • PA (parsing accuracy): 85.05% vs Drain3 58.06%
  • Reproducible: docs/repro_manifest.json pins full 64-hex-character SHA-256 digests, byte sizes, per-dataset line counts, and 7 snapshot sha256s; scripts/verify_repro_manifest.py checks all 11 entries.

Metric scope: PA 85.05% is Loghub Content (timestamps/hosts/levels stripped, Drain protocol); the same implementation on raw input is 31.08%. Same PA metric and annotated-line denominator, evaluated under different input protocols (Loghub Content vs raw); do not mix the two results.

What I will not claim

  • Reproducible ≠ state of the art. There is no leaderboard rank and no third-party run yet.
  • The LLM comparison is a 2,000-line-per-dataset sample, not full data.
  • The HDFS block-level result cannot be re-run here (the 1.5 GB raw log is not on this machine); merged recall is 37.17% — that is the number most worth doubting.

Want to reproduce it? (please do)

Post back three things: OS + Python version, exit code, last 20 lines on failure. A failure is worth more than a pass — it means the docs or environment assumptions are wrong.

Full reproduction (optional, ≈445 MB / 11 entries)

Clone the repo, drop the Loghub datasets under data/ as described in docs/REPRODUCIBILITY.md §5.2, then:

python scripts/verify_repro_manifest.py   # all 11 PASS before continuing
python scripts/refresh_benchmarks.py      # ~60 min
python scripts/prove.py                   # every public claim re-checked

Without data, verify_repro_manifest.py reports 11 MISS and exits 1 — that is expected.

Chinese

Full documentation, design notes and stage-by-stage evidence are in the Chinese README.md.

🙏 External reproducers

Every number in this repository is currently self-reported by the author. This table only lists people who actually ran it (rules: docs/REPRODUCIBILITY.md, section 8):

# Environment Result Date
— none yet be the first —

To get listed: run the install, run, and verify commands above, then reply in Issue #IKWWD3 with OS + Python version + exit codes. Named or anonymous, pass or fail — both count.

License

MIT.

Metadata

Release files for log-ai-compressor 3.0.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for log-ai-compressor 3.0.5
File Size Uploaded Provenance Actions
log_ai_compressor-3.0.5.tar.gz 1.9 MB Without provenance Details for log_ai_compressor-3.0.5.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for log-ai-compressor 3.0.5
File Interpreter ABI Platform Actions
log_ai_compressor-3.0.5-py3-none-any.whl Python 3 none any Details for log_ai_compressor-3.0.5-py3-none-any.whl

Total release size: 2.2 MB

Release files / log_ai_compressor-3.0.5.tar.gz

Download URL log_ai_compressor-3.0.5.tar.gz
Size 1.9 MB
Tags Source
SHA-256 checksum
How to use checksums
9effb9150fcfdd45180c0a294a94873fc69b2dc5be73223aca7654ddaca0cf80
BLAKE2b-256 checksum
How to use checksums
1c779b879c06aa4fd3160a08072036729d7d8e8c40aef9fa499f70f2131b9f35
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.9

Release files / log_ai_compressor-3.0.5-py3-none-any.whl

Download URL log_ai_compressor-3.0.5-py3-none-any.whl
Size 313.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
5ad2ea16df2fa2228d408728c70a18cb1496c3f8af3aa7c87bbe9fa5fea651b8
BLAKE2b-256 checksum
How to use checksums
209deb722bc8bb5f2b726ce8efd59f50f513940199adacb5f1b9bd2480bdbb78
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.9

Security for log-ai-compressor 3.0.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

3.0.5 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page