log-ai-compressor
Every finding carries line numbers + the sha256 of the cited text. Change the cited text — line endings normalized — and verify returns TAMPERED.
For ops / backend / security engineers reading logs. Analysis runs locally — 0 API calls, logs stay on your machine. The optional AI interpretation layer is off by default; if you enable it, only redacted excerpts are sent to the provider you configure, with preview and explicit consent before sending.
verify re-reads the raw log and checks every quote against the cited text. If the cited text was edited, it exits 1 — you never have to trust the tool. Line endings (CR/LF) are normalized per the stated contract and do not count as tampering.
Metric scope: PA 85.05% uses Loghub
Content(timestamps/hosts/levels stripped, Drain protocol); the same implementation on raw input scores 31.08%. Same PA metric and annotated-line denominator, evaluated under different input protocols (Loghub Content vs raw); do not mix the two results.
$ log-ai-compressor verify --report examples/app_report.json
模式:复核已导出报告
对象:报告 app_report.json
证据:共 4 条引用,匹配 3 条,失配 1 条,源不可读 0 条,缺行号 0 条,缺摘要 0 条
自校验结论:TAMPERED # 退出码 1
✗ MISMATCH c3801f905159 examples\app.log 期望 bc5b93cec7cde01c 实际 1b8242c1671f1e29
Key lines from the run (Chinese CLI; stdout + stderr, so the MISMATCH line
may appear before or after the verdict): 4 quotes checked, 3 matched,
1 mismatched → verdict TAMPERED, exit 1. The full run also ends with a one-line
explanation. Hashes and exit codes are byte-identical to what you will get;
reproduce it with the cited-text edit below.
▶ Install & self-proof below · 中文 README
$ python -m pip install -i https://pypi.org/simple "log-ai-compressor==3.0.5"
$ python -c "import log_ai_compressor, pathlib, shutil, os; os.makedirs('examples', exist_ok=True); shutil.copy(pathlib.Path(log_ai_compressor.__file__).parent/'data/examples/app.log', 'examples/app.log')"
$ log-ai-compressor run examples/app.log -f json -o examples/app_report.json
$ log-ai-compressor verify --report examples/app_report.json
自校验结论:VERIFIED
The last line is verbatim output (Chinese CLI): verdict VERIFIED, all 4 quotes
matched, exit 0. No clone needed: line 2 copies the sample log out of the
installed wheel. The key lines at the top are from a real run too: change the cited
db-primary to db-primarY in examples/app.log, re-run verify, and you get
exactly that. Change it back → VERIFIED again. There is no "mostly passed" tier.
Repository is hosted on Gitee (reachable worldwide, but slow from some regions). The package itself installs from PyPI with no dependency on Gitee. Screenshots live in the Chinese README — this page stays text-only so every line renders anywhere.
Open the UI (the recommended entry point)
| Your machine | One command |
|---|---|
| Windows | double-click start.bat in the repo root — finds Python, installs what is missing, opens the browser |
| anything else | pip install "log-ai-compressor[web]==3.0.5" && log-ai-compressor web |
Three steps to a conclusion: ① drag a log into the window (or pick a local file
with Browse) → ② levels default to ERROR/FAIL/WARN only → ③ press Start. The
first thing you get is one card — located root cause, or "insufficient evidence"
with the exact gap — not a wall of template clusters. Press Re-check evidence
and the tool re-reads the raw log line by line and returns VERIFIED / TAMPERED.
The UI is zero-build and zero-CDN, so it works offline; the file browser picks local paths and never uploads. The one exception is FastAPI's own
/api/docs(Swagger), which pulls its stylesheet from jsdelivr — the main screen is unaffected.
CLI (no UI: pipes, CI, agents)
The install and sample commands are in the first screen above. After them:
log-ai-compressor run examples/app.log -f summary
Pin the version: mirrors lag by days, and 2.13.1 (an older release) has no
-f summary. Samples ship inside the wheel (log_ai_compressor/data/examples/);
if examples/ is absent, run / compare fall back by basename and print a
notice. Verbatim output of the command above (the CLI prints Chinese —
numbers, fingerprint and hashes are byte-identical to what you will get):
【日志分析摘要】examples\app.log
原始日志 ≈379 tokens → 摘要 ≈234 tokens(压缩 38%)
确定性指纹 `48e5781aafc4b27fe0f68c5847afe6d891936ac9f2e0af59aefaa9860f9c2886`(同一份日志跑两遍必须完全一致)
总行数 24,错误 8 行,去重后 4 种,时间范围 2024-03-15 08:00:00 ~ 2024-03-15 08:00:25。
已定位根因:java.net.ConnectException: Connection refused(另有 1 个根因候选)
(证据出处末尾 16 位十六进制 = 该段原文 sha256 前 16 位,可自行复核)
1. [P0 | ERROR | ×3 | 根因 | 行 4~13] authorization failed: connection refused to db-primary:5432
证据出处:L4 bc5b93cec7cde01c
-> 因果链源头(1 个错误由其衍生)
2. [P0 | ERROR | ×3 | 行 6~11] request 8812 failed: upstream unavailable
证据出处:L6 21d12907c3115d34
3. [P0 | ERROR | ×1 | 根因 | 行 18~22] java.net.ConnectException: Connection refused
证据出处:L18-L22 68af24f230baff5c
-> Caused-by 因果链直连,根因:java.net.ConnectException: Connection refused
4. [P0 | ERROR | ×1 | 行 23~23] stock check timeout after 3000ms item=SKU-7781
证据出处:L23 a72b9f38845a14ed
Prove it to yourself: edit the cited text
log-ai-compressor run examples/app.log -f json -o examples/app_report.json
log-ai-compressor verify --report examples/app_report.json # VERIFIED, exit 0
Now change db-primary to db-primarY in examples/app.log and re-run the
second command. Key lines (Chinese CLI; the MISMATCH line is stderr, so its
order relative to the verdict can vary; the full run ends with a one-line
explanation):
模式:复核已导出报告
对象:报告 app_report.json
证据:共 4 条引用,匹配 3 条,失配 1 条,源不可读 0 条,缺行号 0 条,缺摘要 0 条
自校验结论:TAMPERED
✗ MISMATCH c3801f905159 examples\app.log 期望 bc5b93cec7cde01c 实际 1b8242c1671f1e29
Translation: 4 quotes checked, 3 matched, 1 mismatched → verdict TAMPERED,
exit 1. Change it back → VERIFIED again. No "mostly passed" tier.
Machine-readable: add --json and read checked: 4 / matched: 3 / mismatched: 1.
What it is (and what it is not)
| grep / tail | Drain3 | 3-shot LLM | this tool | |
|---|---|---|---|---|
| What you get | matching lines | templates | templates | conclusion + line range + sha256 |
| Model needed | no | no | yes | no (core analysis: 0 API calls) |
| Full 10-set GA | — | 71.58% | not run | 88.68% |
| Full 10-set PA | — | 58.06% | not run | 85.05% |
| Output you can re-check | yes | no | no | yes (verify re-reads the log) |
Do the numbers hold up?
Full Loghub 2.0, ten datasets: 1,940,760 lines / 1,621 clusters.
- GA (grouping accuracy): 88.68% vs Drain3 71.58% — same tokenization, same scoring
- PA (parsing accuracy): 85.05% vs Drain3 58.06%
- Reproducible:
docs/repro_manifest.jsonpins full 64-hex-character SHA-256 digests, byte sizes, per-dataset line counts, and 7 snapshotsha256s;scripts/verify_repro_manifest.pychecks all 11 entries.
Metric scope: PA 85.05% is Loghub
Content(timestamps/hosts/levels stripped, Drain protocol); the same implementation on raw input is 31.08%. Same PA metric and annotated-line denominator, evaluated under different input protocols (Loghub Content vs raw); do not mix the two results.
What I will not claim
- Reproducible ≠ state of the art. There is no leaderboard rank and no third-party run yet.
- The LLM comparison is a 2,000-line-per-dataset sample, not full data.
- The HDFS block-level result cannot be re-run here (the 1.5 GB raw log is not on this machine); merged recall is 37.17% — that is the number most worth doubting.
Want to reproduce it? (please do)
Post back three things: OS + Python version, exit code, last 20 lines on failure. A failure is worth more than a pass — it means the docs or environment assumptions are wrong.
- Repo: https://gitee.com/hu-chenyu-dev/log-ai-compressor
- Release v3.0.5: https://gitee.com/hu-chenyu-dev/log-ai-compressor/releases/tag/v3.0.5
- Reproduction guide:
docs/REPRODUCIBILITY.md - Real-reproduction invitation: https://gitee.com/hu-chenyu-dev/log-ai-compressor/issues/IKWWD3
- PyPI: https://pypi.org/project/log-ai-compressor/
Full reproduction (optional, ≈445 MB / 11 entries)
Clone the repo, drop the Loghub datasets under data/ as described in
docs/REPRODUCIBILITY.md §5.2, then:
python scripts/verify_repro_manifest.py # all 11 PASS before continuing
python scripts/refresh_benchmarks.py # ~60 min
python scripts/prove.py # every public claim re-checked
Without data, verify_repro_manifest.py reports 11 MISS and exits 1 — that is expected.
Chinese
Full documentation, design notes and stage-by-stage evidence are in the Chinese
README.md.
🙏 External reproducers
Every number in this repository is currently self-reported by the author. This table only lists people who actually ran it (rules: docs/REPRODUCIBILITY.md, section 8):
| # | Environment | Result | Date |
|---|---|---|---|
| — | none yet | be the first | — |
To get listed: run the install, run, and verify commands above, then reply in Issue #IKWWD3 with OS + Python version + exit codes. Named or anonymous, pass or fail — both count.
License
MIT.
Metadata
Release files for log-ai-compressor 3.0.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| log_ai_compressor-3.0.5.tar.gz | 1.9 MB | Without provenance | Details for log_ai_compressor-3.0.5.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| log_ai_compressor-3.0.5-py3-none-any.whl | Python 3 | none | any | Details for log_ai_compressor-3.0.5-py3-none-any.whl |
Total release size: 2.2 MB
Release files / log_ai_compressor-3.0.5.tar.gz
| Download URL | log_ai_compressor-3.0.5.tar.gz |
|---|---|
| Size | 1.9 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9effb9150fcfdd45180c0a294a94873fc69b2dc5be73223aca7654ddaca0cf80
|
|
BLAKE2b-256 checksum How to use checksums |
1c779b879c06aa4fd3160a08072036729d7d8e8c40aef9fa499f70f2131b9f35
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.9
|
Release files / log_ai_compressor-3.0.5-py3-none-any.whl
| Download URL | log_ai_compressor-3.0.5-py3-none-any.whl |
|---|---|
| Size | 313.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
5ad2ea16df2fa2228d408728c70a18cb1496c3f8af3aa7c87bbe9fa5fea651b8
|
|
BLAKE2b-256 checksum How to use checksums |
209deb722bc8bb5f2b726ce8efd59f50f513940199adacb5f1b9bd2480bdbb78
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.9
|
Security for log-ai-compressor 3.0.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.