LogSpecter

Schema-aware secret scanner for cloud logs. Regex finds candidates; Shannon entropy and a heuristic layer decide whether they are real keys; a JSON-structure layer tells you who leaked what through which field. Streams tens of gigabytes with a flat memory ceiling.

$ logspecter scan cloudtrail-2026-08-30.json.gz --stats

  CRITICAL   openai-api-key   AWS IAM User (Alice) → action: AssumeRole
                              → requestParameters.headers.Authorization
                              cloudtrail-2026-08-30.json.gz:81421 @byte 24118904
                              sk-p********kAyS (len=64)   conf 1.00 / H 5.19

Not "a suspicious string on line 800". The actual identity, the actual API call, the actual JSON path.

Real output

logspecter scan examples/app.log on the bundled sample (run on Windows, first 17 lines; the CLI UI is currently in Chinese):

LogSpecter v0.1.0 · 61 条规则 · JSON 后端 orjson · 分块 4.0 MiB
┌──────────────────────── LogSpecter v0.1.0 扫描摘要 ─────────────────────────┐
│ 唯一泄露  4 条（总出现 4 次）                                               │
│ 等级分布   critical × 2   high × 2                                          │
│   扫描量  1 个来源 / 1 个块 / 2.6 KiB / 18 行                               │
│     性能  0.02s · 0.2 MiB/s · 1 进程                                        │
│ 内存峰值  单进程最高 32.6 MiB（全部进程合计 32.6 MiB）                      │
└─────────────────────────────────────────────────────────────────────────────┘
 CRITICAL  private-key-pem-header  ×1  conf 1.00  H 3.38
  detail
  examples\app.log:13  ----********---- (len=31)

 CRITICAL  stripe-live-secret-key  ×1  conf 1.00  H 4.86
  examples\app.log:6:91
  examples\app.log:6  sk_l********aJ3s (len=33)

 HIGH  authorization-header-basic  ×1  conf 1.00  H 4.50

Why another secret scanner

Most log scanners are a pile of regexes. That fails in production for three reasons, and LogSpecter attacks each one directly.

1. Regex alone cannot tell a key from a session token

A rule like "base64 string of 32+ chars" fires on every pagination cursor, every trace ID, every base64-encoded JSON blob in your logs. LogSpecter runs a second stage on every candidate:

Every decision is recorded on the finding, so you can audit why something was reported or dropped:

"evidence": ["entropy=5.61/6.00(base64url)", "charset_coverage=0.85",
             "non-linguistic", "keyword-nearby", "also-matched:authorization-header-bearer"]

Entropy thresholds are per rule, not global. Authorization: Basic intentionally disables the decode-back check (Basic auth is base64 text); database URLs relax entropy to catch weak human passwords while relying on the scheme://user:pass@host shape for precision.

2. Treating logs as plain text throws away all the context

LogSpecter parses records with orjson (Rust-backed) and recognises the schema it is looking at:

• AWS CloudTrail — IAM identity, eventName, region, source IP, account
• GCP Cloud Logging — principalEmail, methodName, resource, project
• Kubernetes audit — user, verb, objectRef
• Azure Activity Log — identity claims, operationName, result
• Elastic Common Schema, Logback / Log4j2 JSON (including MDC)

Nested JSON-inside-a-JSON-string is expanded too, because requestParameters and MDC fields are full of it. You get a precise path (protoPayload.request.credential) instead of a line number, plus the actor and action needed to actually respond to the incident.

3. readlines() on a 30 GB log is an OOM

The input layer plans line-aligned byte ranges without reading the file (a few seeks and small tail reads), then hands one range per worker. Compressed files and stdin go through a producer/consumer path with a bounded submission window. Resident memory is a function of chunk size, never of file size:

Same memory for 1 GiB as for 256 MiB. Measured with GetProcessMemoryInfo / /proc/self/status and reported by --stats — not a claim, an output field. Numbers from logspecter benchmark on 8 cores / Windows / CPython 3.13, scanning a synthetic mix of CloudTrail records, application logs, and high-entropy-but-harmless noise. Reproduce with logspecter benchmark --size 1GB -j 8; pure-Python throughput is CPU-bound, so expect it to track your single-core speed times the worker count.

Install

pip install logspecter

From source:

git clone https://github.com/Jeffy123-zhu/logspecter
cd logspecter
pip install -e ".[dev]"

Requires Python 3.10+. Runtime dependencies: typer, rich, PyYAML, orjson.

Usage

# a file, a directory, a compressed archive
logspecter scan /var/log/app.log
logspecter scan /var/log/ --recursive
logspecter scan cloudtrail-2026-08-30.json.gz

# a pipe
kubectl logs deploy/api --since=1h | logspecter scan -
aws logs tail /aws/lambda/api --format short | logspecter scan -

# CI gate: fail only on new critical leaks
logspecter scan ./logs --baseline .logspecter-baseline.json --fail-on critical

# machine-readable output
logspecter scan ./logs -f json -o findings.json
logspecter scan ./logs -f csv  -o soc2-evidence.csv
logspecter scan ./logs -f sarif -o results.sarif   # GitHub code scanning

Exit codes: 0 clean, 1 findings at or above --fail-on (default high), 2 bad input.

Options worth knowing

Other commands

logspecter rules list                        # 64 built-in rules across 7 packs
logspecter rules show aws-secret-access-key  # pattern, entropy gate, prefilter
logspecter rules validate ./my-rules.yaml    # lint custom rules
logspecter selftest                          # 64 positive + 25 negative samples
logspecter benchmark --size 1GB -j 8         # throughput and memory on your box

GitHub Action

# .github/workflows/secret-scan.yml
on: [push, pull_request]
permissions:
  contents: read
  security-events: write   # only needed for the SARIF upload
jobs:
  logspecter:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: Jeffy123-zhu/logspecter@v1
        with:
          path: logs/
          format: sarif
          output: logspecter.sarif
          fail-on: high          # critical | high | medium | low | info | none
      - uses: github/codeql-action/upload-sarif@v3
        if: always()
        with:
          sarif_file: logspecter.sarif

Inputs: path (default .), format (table/json/csv/sarif), output, fail-on (default high), baseline, args (extra logspecter scan flags), python-version. Output: exit-code.

pre-commit

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/Jeffy123-zhu/logspecter
    rev: v1.0.0   # or any newer tag
    hooks:
      - id: logspecter

How it compares

Only differences we could verify are filled in; — means "not checked", not "no".

Custom rules

Rules are plain YAML. A rule with the same id as a built-in one overrides it, which is the recommended way to retune thresholds for your environment.

version: 1
pack: acme

rules:
  - id: acme-internal-token
    name: ACME Internal Service Token
    severity: critical
    confidence: high
    pattern: '\bacme_(?:live|prod)_([A-Za-z0-9]{40})\b'
    capture: 1
    tags: [acme, internal]
    entropy:
      min_entropy: 4.4
      min_normalized: 0.72        # entropy ÷ log2(charset size)
      min_length: 40
      min_charset_coverage: 0.6   # unique chars ÷ achievable unique chars
      reject_encoded_text: true

  - id: acme-mdc-secret
    name: Secret in ACME MDC field
    severity: high
    pattern: '\A\s*(\S{12,4096})\s*\Z'
    capture: 1
    json_keys: [acme_token, acme_signature]   # only applied to these JSON keys
    entropy:
      min_entropy: 3.5

logspecter rules validate acme.yaml
logspecter scan ./logs --rules acme.yaml

Full field reference: docs/rules.md.

Library use

from logspecter import engine
from logspecter.rules import load_ruleset
from logspecter.scanner import ScanOptions

config = engine.ScanConfig(ruleset=load_ruleset(), options=ScanOptions())
result = engine.scan(["/var/log/app.log"], config, workers=4)

for group in result.groups:
    f = group.representative
    print(f.severity.value, f.rule_id, f.context_summary(), f"×{group.occurrences}")

print(result.stats.throughput_mb_s, result.stats.peak_rss_max_process)

Accuracy

logspecter selftest runs the bundled corpus with every rule enabled:

检出率 64/64  ·  负样本零误报 25/25

64 positive samples (one per rule, generated from a fixed seed — no real credentials in the repo) and 25 negative samples drawn from the shapes that actually fool regex-only scanners: UUID request IDs, git SHAs, base64-encoded JSON cursors, camel-case class names, template placeholders, AKIAIOSFODNN7EXAMPLE, ISO timestamps, CSS colours, service-account token paths, and unkeyed SHA-256 digests. Each of these is a distinct rejection reason in the entropy layer, and each is a regression test.

--stats reports the funnel on your own data, so the numbers are yours rather than ours:

降噪  正则候选 1,245 → 熵值/上下文层拦下 16 条（1.3%）

Design notes

logspecter/
├── ingest.py      byte-range planning, mmap window reads, gz/bz2/xz/stdin streaming
├── engine.py      chunk scheduling, bounded-window multiprocessing, line-number prefix sums,
│                  fingerprint aggregation
├── prefilter.py   regex AST → mandatory literals + prefix widths, trie merging, screen tree
├── scanner.py     the detection pipeline
├── entropy.py     Shannon entropy and the heuristic gate
├── rules.py       YAML loading, validation, bytes compilation
├── cloud.py       cloud log schema detection and context extraction
├── structured.py  orjson parsing and JSON flattening
├── report/        Rich console, JSON, CSV, SARIF
└── rules/*.yaml   built-in rule packs

Two details that are easy to get wrong and are worth knowing about:

Line numbers under multiprocessing. Workers only know their offset in the file, so they report a chunk-local line number plus that chunk's total line count. The parent computes a prefix sum over chunk line counts and rewrites the findings. No pre-pass over the file, exact file:line.

Report readability. One leaked key repeated 50 000 times is one finding, not 50 000 rows. Findings are aggregated by SHA-256(rule_id ‖ secret)[:16] with an occurrence count and a few sample locations. Overlapping rules on the same value collapse to the most specific one (openai-api-key beats authorization-header-bearer beats sensitive-json-key-value), with the others preserved in the evidence chain.

Security

Reports are redacted by default: masked value plus real length, never plaintext, unless you pass --show-secrets. Baseline files store only fingerprints. The scanner makes no network calls.

If you find a vulnerability, please open a private security advisory rather than a public issue.

Contributing

See CONTRIBUTING.md for the full guide (setup, adding rules, PR flow), and the good first issue label for a place to start.

pip install -e ".[dev]"
pytest              # 360 tests
ruff check .
logspecter selftest

New rules need a positive sample in src/logspecter/samples.py; the test suite fails if any rule lacks one, and tests/test_prefilter.py will tell you if your pattern defeats the prefilter.

Chinese documentation: README.zh-CN.md.

License

Apache-2.0.