pip install loglensai

Why LogLens AI

Most log platforms give you a score and a bill. LogLens AI gives you answers - and runs entirely on your own machine. It's the only log anomaly detector with published, reproducible F1, plus explainable, grouped incidents, live watching, one-line self-alerting for your own apps, and an optional AI root-cause layer.

In one line: the only log anomaly detector with published, reproducible F1 - free, local, explained, grouped into incidents, and able to watch and alert on your services in real time.

🚀 Install

With pip (recommended):

pip install loglensai            # fast + turbo detection, watch, alerts, SDK, RCA
pip install "loglensai[deep]"    # + neural (transformer) semantic mode

With Docker (nothing to install, multi-arch amd64 + arm64):

docker run --rm -v "$PWD:/data" loglensai/loglens analyze --source app.log

Requires Python 3.10+. MIT licensed.

⚡ Quick start

# analyze any log file - format auto-detected, incidents grouped
loglens analyze --source app.log

# maximum throughput on huge files
loglens analyze --source app.log --turbo

# neural semantic mode - best precision
loglens analyze --source app.log --deep

# full incident workflow: turbo scan + AI root-cause + offline HTML report
loglens analyze --source app.log --turbo --rca --html report.html

# watch a running service live - get ONLY the problems, instantly
loglens watch "docker logs -f my-api"

# ask your logs a question in plain English
loglens ask "why did the payment service start timing out?" --source app.log

# reproducible accuracy benchmark against labeled data
loglens benchmark labeled.log --min-f1 0.90

Add Sentry-style self-alerting to your own app in one line:

import loglens
loglens.init(app_name="checkout-api")   # → Slack / Teams / Email on serious events

📖 Full command & SDK reference → loglensai.com/docs and DOCUMENTATION.md.

📊 Benchmarks - measured, reproducible, honest

All numbers on real labeled datasets from Loghub. Reproduce them yourself → BENCHMARK.md.

Accuracy - Loghub BGL (500,000 lines, 206,847 labeled alerts)

• 🎯 Zero missed alerts - 1.000 recall across all 206,847 alerts, every mode.
• 🧠 Deep mode measurably beats the baseline - semantic embeddings cut false positives ~18%. Provable AI value, not marketing.
• 🚀 Turbo matches fast-mode accuracy exactly at higher throughput - speed with no accuracy tradeoff.

Generality - no retuning (Sandia Thunderbird, 500k all-normal lines)

Needle-in-a-haystack - 30/30 injected incidents caught

Kernel panic, OOM, disk failure, security breach and data corruption injected into routine logs across 6 formats (Apache, Spark, HDFS, HealthApp, OpenStack, Thunderbird): 100% recall, zero configuration.

✨ Features

🔬 Detection core

• Three engines, one unified score - fast (from-scratch statistical: TF-IDF template embeddings, weighted density clustering, severity/rarity/chronic scoring - no ML libs in the core), turbo (same accuracy, parallel byte-range scanning + template dedup), and deep (transformer semantic embeddings that understand log meaning, run on unique templates for speed).
• Incident families - repeated anomalies collapse into one incident with an ×N count. No scrolling through 200 identical errors.
• Explainable by default - every flag ships with a plain-language reason (rare + severe + burst context), not just a number.
• 10+ log formats auto-detected - Apache, Linux, Mac, HDFS, Spark, Zookeeper, OpenStack, Thunderbird, BGL, HealthApp & generic. No config, ever.
• Flexible ingestion - files, URLs, and live commands.

📡 Live & always-on

• loglens watch - point it at docker logs -f, kubectl logs -f, or journalctl -f and it prints only the problems, instantly. CRITICAL/FATAL surface immediately; Ctrl-C prints a summary (optionally with AI root-cause + HTML dashboard).
• Self-alerting in one line (loglens.init) - Sentry-style alerts to Slack / Teams / Email the moment something serious happens, including uncaught crashes. De-duplicated, rate-limited, sent from a background thread so it never risks your app. Works with zero AI setup (built-in cause hints) and gets richer with a BYO LLM key.

🤖 AI layer (bring your own key)

• AI root-cause analysis (--rca) - BYO key (OpenAI / Azure / Groq). Sends only grouped anomaly summaries to the LLM - never your full log - so it's cheap, private, and coherent.
• Natural-language Q&A (loglens ask) - ask "why did db-service degrade?" and get an answer grounded in the detected anomalies.

🐍 Python SDK

from loglens import analyze
result = analyze("app.log")                 # or lines=[...], cmd="docker logs api"
for a in result.anomalies:
    print(a.level, a.score, a.message, a.reasons)
print(result.rca().report)                  # AI root-cause (BYO key)

• analyze() / analyze_async() - "here are logs, give me the problems."
• LogLensHandler - drop into Python's logging so your app raises its own alarm.
• LiveDetector - feed a custom stream line-by-line, get anomalies out (powers watch).
• .rca(), .ask(...), .save_html(...), .save_rca(...) on any result or live session.

📈 Reporting & benchmarking

• Self-contained HTML report (--html) - dark-themed dashboard, severity + per-service breakdown, score distribution. Fully offline (no CDN), embeds the RCA narrative.
• Speed benchmark (loglens bench) - lines/sec, time-to-insight, peak RAM across modes.
• Reproducible accuracy benchmark (loglens benchmark) - precision / recall / F1 on labeled data, grid-search, and a --min-f1 CI gate.
• 100% local & private - detection never leaves your machine; air-gap friendly.

🧭 How it works

1. Parse - streaming parser auto-detects the log format.
2. Template - messages mined into templates; per-template volume statistics.
3. Embed - TF-IDF (fast/turbo) or transformer (deep); one vector per unique template for speed.
4. Detect - an ensemble score blends severity prior, template rarity, embedding distance, chronic-pattern damping, and a global-rarity bonus into a calibrated continuous score.
5. Group - repeated anomalies collapse into incident families (×N).
6. Explain - each anomaly reported with its human-readable reason; optionally an LLM writes the root-cause narrative.
7. Deliver - terminal, live watch, Slack/Teams/Email via init, or offline HTML report.

🐳 Docker

Multi-arch images (linux/amd64 + linux/arm64) on Docker Hub:

# analyze a file (mount the folder that holds it)
docker run --rm -v "$PWD:/data" loglensai/loglens analyze --source app.log

# live-watch another container (mount the docker socket)
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock:ro \
  loglensai/loglens watch "docker logs -f my-api" --rca

Tags: latest, 0.3, 0.3.3 (slim), and deep (adds neural mode).

🗺️ Roadmap

• 🔧 Chronic-noise damping improvements for Linux/Mac daemon logs.
• 📦 Prebuilt GitHub Action for CI log gating.
• 📊 More alert channels - PagerDuty, Opsgenie, generic webhooks.
• 🌐 Optional lightweight web UI for the HTML dashboards.

📜 Honesty notes

• Accuracy measured on Loghub line-level labels (token - = normal).
• Deep mode embeds unique templates only - a real optimization, disclosed.
• --rca, ask, and alert cause-hints send only grouped anomaly summaries to the LLM, never the full log.
• Alerting works fully offline with built-in cause hints; an LLM key only enriches the narrative.
• All results reproducible with the included harness. See BENCHMARK.md.

🤝 Contributing & support

Issues and PRs welcome. If LogLens AI saves you a 2 a.m. page, please ⭐ star the repo - it genuinely helps.

• 🌐 Website: loglensai.com
• 📦 PyPI: pypi.org/project/loglensai
• 🐳 Docker Hub: hub.docker.com/r/loglensai/loglens
• 📖 Docs: loglensai.com/docs

License

MIT - see LICENSE. Use it in production, commercially, anywhere.