logshrink 1.0.0

  ██╗      ██████╗  ██████╗ ███████╗██╗  ██╗██████╗ ██╗███╗   ██╗██╗  ██╗
  ██║     ██╔═══██╗██╔════╝ ██╔════╝██║  ██║██╔══██╗██║████╗  ██║██║ ██╔╝
  ██║     ██║   ██║██║  ███╗███████╗███████║██████╔╝██║██╔██╗ ██║█████╔╝
  ██║     ██║   ██║██║   ██║╚════██║██╔══██║██╔══██╗██║██║╚██╗██║██╔═██╗
  ███████╗╚██████╔╝╚██████╔╝███████║██║  ██║██║  ██║██║██║ ╚████║██║  ██╗
  ╚══════╝ ╚═════╝  ╚═════╝ ╚══════╝╚═╝  ╚═╝╚═╝  ╚═╝╚═╝╚═╝  ╚═══╝╚═╝  ╚═╝

💥 The Problem vs. Solution

When developing or debugging backend microservices, Kubernetes pods, and container clusters:

• Polling Floods: Thousands of identical Kubernetes liveness probes (GET /healthz 200), database heartbeats (SELECT 1), and Prometheus scrapes scroll by every minute.
• Lost Errors: When a real deadlock, panic, or unhandled 500 error occurs, it scrolls off your terminal buffer in 300 milliseconds.
• Brittle Grep Pipelines: Chaining grep -v "health" | grep -v "metrics" is dangerous—it blindly deletes genuine failures occurring on health endpoints and mutilates multi-line tracebacks.

Side-by-Side Terminal Comparison

┌───────────────────────────────────────────────┐   ┌───────────────────────────────────────────────┐
│           BEFORE: Raw Terminal Flood          │   │             AFTER: With LogShrink             │
├───────────────────────────────────────────────┤   ├───────────────────────────────────────────────┤
│ 19:00:01 [INFO] GET /healthz 200 (took 0.4ms) │   │ 19:00:01 [INFO] GET /healthz 200 (took 0.4ms) │
│ 19:00:02 [INFO] GET /healthz 200 (took 0.3ms) │   │   [-] [x480 suppressed] [INFO] GET /healthz   │
│ 19:00:03 [INFO] GET /healthz 200 (took 0.5ms) │   │                       (last seen 19:04:11)    │
│ 19:00:04 [INFO] GET /healthz 200 (took 0.4ms) │   │                                               │
│ ... [480 identical lines flooding buffer] ... │   │ ┌── INCIDENT #1: ERROR ────────────────────── │
│ 19:04:12 [ERROR] Database deadlock!           │   │ │ 19:04:12 [ERROR] Database deadlock!         │
│ Traceback (most recent call last):            │   │ │ Traceback (most recent call last):          │
│   File "/app/db.py", line 42, in query        │   │ │   File "/app/db.py", line 42, in query      │
│     raise OperationalError("deadlock")        │   │ │     raise OperationalError("deadlock")      │
│ 19:04:13 [INFO] GET /healthz 200 (took 0.3ms) │   │ └──────────────────────────────────────────── │
│ 19:04:14 [INFO] GET /healthz 200 (took 0.4ms) │   │ 19:04:13 [INFO] GET /healthz 200 (took 0.3ms) │
│                                               │   │                                               │
│   [CRITICAL ERROR SCROLLED OFF BUFFER]        │   │   [NOISE COLLAPSED • ERROR BOXED • 0 DRIFT]   │
└───────────────────────────────────────────────┘   └───────────────────────────────────────────────┘

LogShrink solves this with zero configuration. It dynamically aggregates repetitive polling noise into live counter badges like [-] [x480 suppressed] while preserving the first occurrence, auto-boxes multi-line stack traces (Python, Go, Java, Rust, Node.js), and exports incident postmortems straight to Markdown or your system clipboard.

📊 Feature Comparison

Why use LogShrink instead of standard Unix utilities or heavyweight observability platforms?

⚡ Quickstart & Production Recipes

LogShrink behaves like a standard Unix pipe: pipe any command into logshrink, or pass files directly.

🐳 Docker & Docker Compose

# Follow all services in Docker Compose
docker compose logs -f --tail=100 | logshrink

# Strip Docker container prefixes and follow specific container
docker logs -f my-api-backend | logshrink -p docker

☸️ Kubernetes Pods & Deployments

# Live stream logs across all replicas of a deployment
kubectl logs -f deployment/api-gateway --tail=50 --max-log-requests=10 | logshrink

# Filter production pod down to warnings and fatal incidents only
kubectl logs -f pod/payment-worker-8d76b -n production | logshrink --errors-only

🐧 Systemd Journal & Linux Daemons

# Nginx or Caddy web server
journalctl -u nginx.service -f -n 200 | logshrink -p web

# Follow any system service cleanly
journalctl -u backend.service -f -o cat | logshrink

🚀 Backend Frameworks

# Python: FastAPI / Uvicorn / Flask
uvicorn app.main:app --reload | logshrink

# Python: Django / Gunicorn
gunicorn config.wsgi:application --access-logfile - | logshrink -p web

# Node.js: Express, NestJS, Next.js
npm run dev | logshrink

# Go: Gin, Fiber, Chi
go run ./cmd/server | logshrink

# Rust: Axum, Actix-web, Tokio
cargo run | logshrink

📋 Instant Incident Report for Slack & GitHub

When an unexpected crash occurs, generate an instant postmortem Markdown report and copy it to your clipboard with a single command:

logshrink /var/log/app.log --export incident_report.md --copy

Generated Markdown Output (incident_report.md):

# 🚨 LogShrink Incident & Noise Reduction Report

- **Generated At**: `2026-10-13 19:04:15`
- **Source**: `/var/log/app.log`
- **Total Lines Processed**: `14,820`
- **Noise Suppressed**: `14,350 (96.8%)`
- **Unique Incidents Detected**: `1`

## 📊 Incident Overview

| ID | Level | Occurrences | First Seen | Last Seen | Summary |
|---|---|---|---|---|---|
| #1 | **ERROR** | 1x | `19:04:12` | `19:04:12` | `ERROR: Database deadlock detected!` |

---

## 🔍 Detailed Incident Logs

### Incident #1 — ERROR (1x)
- **First Seen**: `19:04:12`
- **Last Seen**: `19:04:12`

```log
2026-10-13 19:04:12 [ERROR] Database deadlock detected!
Traceback (most recent call last):
  File "/app/db.py", line 42, in query
    raise OperationalError("deadlock detected")

---

<a id="demo"></a>
## 🖥️ Live Terminal Walkthrough

Test LogShrink immediately with the included synthetic demo stream:

```bash
python -m logshrink demo.log

2026-10-13 19:00:00.123 [INFO] App initialized on port 8080
2026-10-13 19:00:01.001 [INFO] GET /healthz 200 OK (took 0.4ms)
  [-] [x5 suppressed] [INFO] GET /healthz 200 OK (last seen 23:18:49)
▲ 2026-10-13 19:00:06.100 [WARN] High memory consumption detected: 82% threshold reached
2026-10-13 19:00:07.001 [INFO] GET /healthz 200 OK (took 0.4ms)
  [-] [x2 suppressed] [INFO] GET /healthz 200 OK (last seen 23:18:49)
┌── INCIDENT #1: ERROR ─────────────────────────────────────────────────
│ 2026-10-13 19:00:09.500 [ERROR] Failed to connect to Redis cache cluster at 10.0.0.45:6379
│ Traceback (most recent call last):
│   File "/app/cache/redis.py", line 48, in connect
│     sock.connect((self.host, self.port))
│   File "/usr/local/lib/python3.12/socket.py", line 710, in connect
│     raise ConnectionRefusedError(errno.ECONNREFUSED, 'Connection refused')
│ ConnectionRefusedError: [Errno 111] Connection refused
└───────────────────────────────────────────────────────────────────────
2026-10-13 19:00:10.001 [INFO] Fallback to in-memory cache activated
2026-10-13 19:00:11.001 [INFO] GET /healthz 200 OK (took 0.2ms)
  [-] [x3 suppressed] [INFO] GET /healthz 200 OK (last seen 23:18:49)
─── LogShrink Summary ────────────────────────────────────
  Total lines processed : 20
  Lines passed through  : 13
  Lines suppressed      : 7 (35.0% noise reduced)
  Incidents captured    : 1
  Processing time       : 0.00s (7,263 lines/sec)
──────────────────────────────────────────────────────────

🤖 GitHub Actions CI Integration

Deduplicate noisy build outputs and automatically render clean incident postmortems in your GitHub Actions workflow summary in 3 lines of YAML:

- name: Clean & Summarize CI Logs
  uses: umutgungorr/logshrink@v1
  with:
    log-file: ci-test-run.log

🌟 Why Use LogShrink in GitHub Actions?

• No More CI Truncation: GitHub Actions truncates browser log viewers when output exceeds buffer limits. LogShrink suppresses repetitive polling noise (e.g. health checks, heartbeat pings) while preserving every unique error.
• Native Step Summary: Automatically appends a rich Markdown incident card to $GITHUB_STEP_SUMMARY, so your team sees root-cause stack traces without downloading raw artifacts.
• Cross-Platform Runner Support: Runs seamlessly on ubuntu-latest, macos-latest, and windows-latest.
• Lightning Fast (< 2s): Zero external dependencies means instant setup and execution in your pipeline.

📋 Full Production Workflow Recipe

name: Test Suite & Service Checks

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Run Integration Tests & Save Log
        id: run-tests
        run: |
          pytest -v 2>&1 | tee test-output.log
        continue-on-error: true

      - name: Process Logs with LogShrink
        id: logshrink
        uses: umutgungorr/logshrink@v1
        with:
          log-file: test-output.log
          summary: 'true'
          export-file: test-incidents.md

      - name: Fail CI if Incidents Detected
        if: steps.run-tests.outcome == 'failure' || steps.logshrink.outputs.incident-count > 0
        run: |
          echo "Detected ${{ steps.logshrink.outputs.incident-count }} critical incidents in test run!"
          exit 1

⚙️ Action Inputs

📤 Action Outputs

📦 Installation

LogShrink has zero external dependencies and works out-of-the-box on Linux, macOS, and Windows with any Python 3.9+ runtime.

⚡ Instant execution with uvx (Zero installation required)

# Run directly from repository
uvx --from git+https://github.com/umutgungorr/logshrink.git logshrink --help

# Pipe anywhere
docker compose logs -f | uvx --from git+https://github.com/umutgungorr/logshrink.git logshrink

🔒 Isolated install via pipx (Recommended for CLI tools)

pipx install git+https://github.com/umutgungorr/logshrink.git

🐍 Standard install via pip

# From GitHub repository
pip install git+https://github.com/umutgungorr/logshrink.git

# Or standard PyPI install (once published)
pip install logshrink

🛠️ Build from source

git clone https://github.com/umutgungorr/logshrink.git
cd logshrink
pip install -e .

⚙️ CLI Reference

usage: logshrink [-h] [-f] [-t THRESHOLD] [-w WINDOW]
                 [-p {auto,docker,json,web,raw}] [-e]
                 [--export PATH] [--copy] [-i REGEX]
                 [--no-color] [--no-stats] [-v]
                 [FILE ...]

Options & Flags

Presets

• auto (default): Dynamically detects JSON lines, Docker compose container prefixes, and standard ISO/Syslog timestamped logs.
• docker: Strips container prefixes (e.g. api-server_1 | ) and container engine run timestamps.
• json: Unpacks structured JSON fields (level, msg, message, status, path) into compact human signatures.
• web: Tailored for HTTP access logs (Nginx, Apache, Caddy, Uvicorn), extracting method, route, and status code.
• raw: Performs strict line-by-line deduplication without fuzzy masking.

🔬 How It Works: Pipeline Architecture

LogShrink processes streaming log events through a high-performance 5-stage pipeline designed for $O(1)$ constant memory consumption:

 ┌────────────────────────────────────────────────────────┐
 │           Incoming Stream (stdin / tail -f)            │
 └───────────────────────────┬────────────────────────────┘
                             │
                             ▼
 ┌────────────────────────────────────────────────────────┐
 │ 1. Ingestion & Preset Tokenizer                        │
 │    • Strips container tags (`backend_1 |`)             │
 │    • Extracts structured JSON fields (`msg`, `level`)  │
 │    • Normalizes Web access patterns (`GET /healthz`)   │
 └───────────────────────────┬────────────────────────────┘
                             │
                             ▼
 ┌────────────────────────────────────────────────────────┐
 │ 2. Dynamic Pattern Normalizer                          │
 │    • Masks ephemeral timestamps (`<TIMESTAMP>`)        │
 │    • Masks UUIDs, IPv4/IPv6, hex addresses (`0x7f...`) │
 │    • Masks durations (`0.4ms`), ports (`:8080`), sizes │
 │    • Computes canonical pattern fingerprint            │
 └───────────────────────────┬────────────────────────────┘
                             │
                             ▼
 ┌────────────────────────────────────────────────────────┐
 │ 3. Multi-Pattern Sliding-Window Aggregator             │
 │    • Maintains separate debouncing state per pattern   │
 │    • Tracks repetition counts within time window       │
 │    • Suppresses lines meeting threshold (default: >=2) │
 └───────────────────────────┬────────────────────────────┘
                             │
                             ▼
 ┌────────────────────────────────────────────────────────┐
 │ 4. Incident Boundary Detector                          │
 │    • Detects multi-line traces (Python, Go, Java,      │
 │      Rust, Node.js async traces)                       │
 │    • Captures HTTP 5xx server exceptions               │
 │    • Filters false positives (`0 errors`, `err: nil`)  │
 └───────────────────────────┬────────────────────────────┘
                             │
                             ▼
 ┌────────────────────────────────────────────────────────┐
 │ 5. Terminal & Incident Exporter                        │
 │    • High-contrast ANSI bounding boxes for incidents   │
 │    • Live suppression counter badges                   │
 │    • Markdown postmortem report generator              │
 │    • Direct system clipboard integration               │
 └────────────────────────────────────────────────────────┘

1. Ingestion & Preset Tokenizer: Fast pre-filtering that strips environment artifacts (such as Docker Compose container names or syslog headers) while preserving the underlying message.
2. Dynamic Pattern Normalizer: Converts transient variable data into stable tokens without requiring complex regexes from the user. Timestamps, request IDs, IP addresses, memory pointers, and microsecond latencies are unified into a canonical signature.
3. Multi-Pattern Sliding-Window Aggregator: Avoids buffer explosions by keeping rolling pattern hashes. If multiple periodic events interleave (e.g., metric scrape A followed by heartbeat B), both are independently tracked and suppressed without losing count.
4. Incident Boundary Detector: Inspects lines for error severity tokens and multi-line exception starters across 5 major programming languages. Blocks false positives such as 0 errors found or err: nil.
5. Terminal & Incident Exporter: Highlights true errors inside ANSI border boxes, renders compact badges for collapsed noise, and produces markdown exports for incident postmortems.

🚀 Benchmarks & Performance

Benchmarked against a 100,000-line synthetic log stream (containing interleaved healthchecks, metrics, warnings, and stack traces):

🧪 Comprehensive Test Suite

LogShrink has 100% passing tests with 84% branch coverage:

# Run complete pytest test suite
pytest

# Run tests with detailed line coverage report
pytest --cov=src/logshrink --cov-report=term-missing

The 59-test suite covers:

• ✅ Multi-language stack trace boundaries (Python, Go, Java, Rust, Node.js)
• ✅ Ephemeral masking for UUIDs, IPv4/IPv6, ephemeral ports, hex pointers, durations, and sizes
• ✅ Interleaved recurring pattern collapsing and debouncing
• ✅ False-positive phrase suppression (err: nil, 0 errors, status=ok error=none)
• ✅ Web access HTTP 5xx error and 4xx warning boundary isolation
• ✅ Docker JSON file driver log payload extraction
• ✅ Real-time follow mode (-f) and file rotation handling
• ✅ Clean broken pipe handling (SIGPIPE) in CLI pipelines
• ✅ System clipboard (pbcopy, wl-copy, xclip, clip.exe) and Markdown export formatters

⚡ Part of the Developer Diagnostics Suite

LogShrink is part of the zero-dependency Developer Diagnostics Suite by @umutgungorr:

• 🌲 PortTree: Dev socket tree scanner, Docker proxy unmasker, and process-tree killer.
• ⚡ LogShrink: Zero-config CLI stream filter that collapses noisy polling logs and auto-boxes stack traces.
• 🩺 RepoPulse: Git forensic health, architectural hotspot churn, and bus-factor risk engine.

🤝 Contributing

Contributions, issues, and feature requests are welcome!

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Run formatting and tests (ruff check . && pytest)
4. Commit your changes (git commit -m 'feat: add amazing feature')
5. Push to the branch (git push origin feature/amazing-feature)
6. Open a Pull Request

Please refer to CONTRIBUTING.md and CODE_OF_CONDUCT.md for detailed guidelines.

📄 License

Distributed under the MIT License. See LICENSE for more information.

Copyright © 2026 Umut Güngör.