open-source-secret-watcher 0.1.1

Defensive-security / responsible-disclosure platform that scans public GitHub repos for exposed secrets.

Open Source Secret Watcher

A defensive-security and responsible-disclosure platform that discovers public GitHub repositories, scans them for potentially exposed secrets using the official TruffleHog CLI, classifies and scores findings (reducing false positives with regex + entropy analysis), and helps repository maintainers remediate.

Built for education, portfolio use, defensive security, and responsible disclosure.

---

⚠️ Safety principles (enforced in code, not just documented)

This tool is deliberately constrained so it cannot be misused as an attack tool:

• Public repositories only.
• It never uses, authenticates with, sends requests using, or tests a discovered credential, and never tries to determine whether one is active.
• The regex/entropy layers only decide whether a string resembles a known credential format. They make no network calls.
• Raw secrets are never stored. Only a masked preview (prefix…last4) and a one-way SHA-256 fingerprint are persisted. There is no config flag and no database column that holds the raw value — the capability does not exist in this build. The raw string lives in memory only long enough to compute the preview/fingerprint and run format checks.
• Raw secrets are never logged.
• Disclosure reports are written to disk for a human to review and send — nothing is emailed or posted automatically.

The intent is to help owners fix leaks, not to collect or exploit credentials.

---

Features

Area	What it does
Discovery	GitHub Search API across configurable keywords, recent-creation window, pagination, rate-limit backoff, concurrent queries. Dry-run by default.
Scanning	Programmatic TruffleHog execution (async), JSON parsing, concurrent scans, retries, scan history, already-scanned tracking.
Classification	OpenAI, Anthropic, AWS, GitHub, Google, Stripe, Slack, Azure, generic, unknown.
Regex validation	Format/length/prefix checks per provider; rejects obvious placeholders; assigns confidence.
Entropy analysis	Shannon entropy with hex/base64-aware thresholds; combined entropy + regex confidence score.
Risk scoring	Critical / High / Medium / Low from category, confidence, exposure location, repo popularity, commit age, and exposure history.
Storage	SQLite + SQLAlchemy. Repo metadata, scan records, findings (masked).
Notifications	Email + Markdown responsible-disclosure templates with remediation guidance. Files only.
Dashboard	Live Rich terminal dashboard.
Export	CSV export of findings (masked).

---

Architecture

discover → queue(DB) → scan(TruffleHog) → parse(JSON) → validate(regex+entropy)
        → score(confidence + risk) → store(DB) → notify(files) / dashboard / export

secret_watcher/
├── config/        # Pydantic settings (TOML + env)
├── discovery/     # GitHub Search API client (+ offline mock) and orchestrator
├── scanner/       # TruffleHog runner (+ offline mock) and scan manager
├── parser/        # TruffleHog JSON → normalised findings
├── validators/    # regex_validator.py, entropy_validator.py, scoring.py
├── database/      # SQLAlchemy models, session, data-access helpers
├── notifications/ # remediation guidance, Jinja2 templates, reporter
├── dashboard/     # Rich terminal dashboard
├── cli/           # Typer CLI
├── security.py    # masking + fingerprint (the only code that touches a raw secret)
└── export.py      # CSV export
tests/             # pytest suite (pure modules + mocked adapters)

Stages communicate through the database, so they are decoupled and resumable. Pure logic (validators, scoring, parser) does no I/O and is unit-tested. Network/process stages sit behind interfaces with offline mocks, so the whole pipeline runs without a token or TruffleHog via --mock.

---

Installation

See INSTALL.md for full details. Quick version:

python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
# Install TruffleHog v3 (for real scans): https://github.com/trufflesecurity/trufflehog
cp config.example.toml config.toml         # edit as needed
export GITHUB_TOKEN=ghp_your_token         # for the GitHub Search API

Requires Python 3.12+.

---

Usage

Every command accepts --config PATH and --mock (offline mocks, no token/TruffleHog).

# 1. Discover public repos (dry-run; add --queue to enqueue them)
secretwatch discover --queue

# 2. Scan the queue with TruffleHog
secretwatch scan

# 3. Generate responsible-disclosure reports (email + markdown) into reports/
secretwatch report

# 4. Live dashboard
secretwatch dashboard            # --once to render a single frame

# 5. Export findings to CSV (masked previews only)
secretwatch export findings.csv

# 6. Aggregate statistics
secretwatch stats

Try it offline (no token, no TruffleHog)

secretwatch discover --mock --queue
secretwatch scan --mock
secretwatch dashboard --mock --once

---

Sample disclosure report

See reports/samples/ for a full email + Markdown example. Excerpt:

  Repository:   https://github.com/example/env-sample
  File:         .env (line 3)
  Category:     AWS credential
  Confidence:   0.34 (CRITICAL risk)
  Reference:    AKIAIOS…MPLE   (masked — the full value is not stored)

Sample dashboard

A captured render is in reports/samples/dashboard.txt:

╭───── Overview ──────╮ ╭─────── By Severity ───────╮ ╭───── By Type ─────╮
│ Repositories  2     │ │ CRITICAL    1             │ │ AWS credential  2 │
│      Scanned  2     │ │ HIGH        3             │ │ Stripe key      2 │
│        Queue  0     │ │ MEDIUM      0             │ ╰───────────────────╯
│     Findings  4     │ │ LOW         0             │
╰─────────────────────╯ ╰───────────────────────────╯
Recent Detections: example/env-sample | Stripe key | HIGH | sk_live…LE12

---

Testing

pytest -q

The suite covers regex/entropy/scoring/parser logic, database helpers, and a full mock-driven pipeline that asserts no raw secret is ever persisted to the database, reports, or CSV.

---

Configuration

All behaviour is driven by config.toml (see config.example.toml). Notable knobs:

• discovery.keywords, discovery.created_within_days, discovery.max_results
• scanner.trufflehog_path, scanner.max_concurrent_scans, scanner.max_retries
• entropy.* thresholds, scoring.* weights and scoring.min_confidence
• reports.reporter_name / reports.reporter_contact for the disclosure signature

The GitHub token is read from GITHUB_TOKEN in the environment, never from the file.

---

License

MIT. Use responsibly. You are responsible for complying with GitHub's Terms of Service and all applicable laws when using this tool.