harpocrates 0.4.0

ML-powered secrets detection tool

Harpocrates

ML-powered secrets detection that catches what regex can't see

---

AI coding tools leak secrets at 2× the rate of human-written code — 3.2% vs 1.6% of commits. 29 million secrets were exposed in 2025, up 34% year over year. One misconfigured environment variable cost a team $87k in a single night.

Regex scanners look for AWS_ACCESS_KEY_ID and GITHUB_TOKEN. They miss client_secret, ENCRYPTION_KEY, and API_SECRET — the names developers actually use. They miss secrets buried in comments. They miss env-var fallbacks. They miss the code your AI coding assistant just generated.

Harpocrates catches what slips through.

---

Harpocrates vs TruffleHog

Benchmarked on a 5,000-line held-out evaluation set, cross-referenced against TruffleHog's output using 1,430 TruffleHog findings as ground truth.

	TruffleHog	Harpocrates
Unique lines flagged	606	1,300
Lines only this tool caught	449	1,143
Lines both tools caught	157	157
Recall (held-out set)	—	97.3%
Precision (held-out set)	—	87.1%
Detection approach	Regex + live API verification	Regex + entropy + ML context
Scans comments	❌	✅
Catches ambiguous variable names	❌	✅
Catches env-var fallback leaks	❌	✅
Catches AI-scaffolded credential patterns	❌	✅
Classifies secret type (password, api_token, jwt…)	❌	✅
Verifies credentials are live via API calls	✅	❌

Harpocrates caught 1,143 lines TruffleHog missed. TruffleHog caught 449 Harpocrates missed.

TruffleHog's edge is live credential verification — it calls the AWS, GitHub, and Stripe APIs to confirm a key is still active. Harpocrates doesn't do that. If a key matches a known format and is still live, TruffleHog will catch it. Run both.

What Harpocrates catches that TruffleHog can't

1. Secrets under ambiguous variable names

TruffleHog requires the variable name to match a known pattern. Harpocrates uses ML to evaluate the token, the variable name, and the surrounding context together.

Code	TruffleHog	Harpocrates
client_secret = "eyJhbGci..."	❌	✅ JWT (0.97)
ENCRYPTION_KEY = "AKIAIOSFODNNabcd1234"	❌	✅ AWS key (0.90)
API_SECRET = "AKIAJOE7MTPSO7EXAM1E"	❌	✅ AWS key (0.80)
ACCESS_KEY = "eyJhbGciOiJSUzI1NiJ9..."	❌	✅ JWT (0.97)
my_token = "xoxb-12345-67890-abcdef..."	❌	✅ Slack token (0.94)

2. Secrets buried in comments

TruffleHog skips comment lines. Harpocrates scans all comment styles — #, //, /* */, <!--, -- — and flags any high-entropy token it finds there.

# old: DB_PASSWORD = "prod_hunter2_xxxSECRET"     ← TruffleHog: skipped. Harpocrates: caught.
// const apiKey = 'sk_live_NnyQ...';               ← TruffleHog: skipped. Harpocrates: caught.
TRACE_ID=$(git log) # AKIAMXWCUIXEKLN02TKZ         ← TruffleHog: sees git command. Harpocrates: sees the key in the comment.

3. Environment variable fallback leaks

A hardcoded fallback in os.getenv() is a secret that ships to production whenever the env var is unset. TruffleHog doesn't model this pattern. Harpocrates does.

SECRET_KEY = os.getenv("SECRET_KEY", "hardcoded_prod_value_xxx")   # ← caught
api_key = config("API_KEY", default="sk_live_realkey123456789")     # ← caught
const key = process.env.KEY ?? "literal_secret_value";             # ← caught

4. Dev/prod swap comments

Rotating from a test key to a production key and leaving the prod key in a comment is a common pattern. The comment leaks the live credential.

STRIPE_KEY = "sk_test_abc"  # PROD: sk_live_xyz_the_real_one    ← caught

5. AI-scaffolded code

AI coding assistants frequently hardcode credentials when scaffolding integrations. Harpocrates is trained on AI-generated code patterns specifically for this.

# Generated by Copilot:
client = OpenAI(api_key="sk-proj-xLMN3kJqPv...")    # ← caught
stripe.api_key = "sk_live_9QnKsF4jQ8M..."           # ← caught

6. Cross-language coverage

The same ML model covers Python, JavaScript/TypeScript, Go, Java, Ruby, YAML, Dockerfile, Terraform, and Helm values files. TruffleHog's pattern library varies by detector; Harpocrates's entropy+ML layer is language-agnostic.

Where TruffleHog has the edge

• Live verification — TruffleHog calls provider APIs to confirm a secret is active. Harpocrates never makes network calls.
• Format-anchored patterns — credentials that match a strict format (GitHub PAT prefix ghp_, Stripe's sk_live_) are caught with zero false positives by TruffleHog's regex. Harpocrates catches these too, but TruffleHog also confirms liveness.

Recommended setup: run Harpocrates as a pre-commit hook (catches the full range before anything reaches git) and TruffleHog in CI (confirms active credentials that made it through).

---

Install

pip install harpocrates
harpocrates scan .

With ML verification (recommended — 87% precision, 97% recall on held-out real-world set):

pip install "harpocrates[ml]"
harpocrates scan . --ml

With REST API server:

pip install "harpocrates[api]"
harpocrates serve

Everything at once:

pip install "harpocrates[all]"

---

Usage

# Scan a directory
harpocrates scan ./my_project

# Scan a single file
harpocrates scan config.env

# Output as JSON (pipe-friendly)
harpocrates scan ./my_project --json

# Enable ML verification to suppress false positives
harpocrates scan ./my_project --ml

# Only fail CI on high or critical findings
harpocrates scan ./my_project --fail-on high

# Ignore specific patterns
harpocrates scan ./my_project --ignore "*.test.js,fixtures/*"

Pre-commit hook

repos:
  - repo: https://github.com/Skipa776/Harpocrates
    rev: v0.2.0
    hooks:
      - id: harpocrates

Add to .pre-commit-config.yaml, then run pre-commit install. Harpocrates scans every staged file before each commit.

---

Configuration

harpocrates scan flags

Flag	Default	Description
--ml	off	Enable ML verification to reduce false positives
--ml-threshold FLOAT	0.19	ML confidence threshold 0.0–1.0. Lower = more recall, higher = more precision
--fail-on LEVEL	medium	Severity that triggers exit code 1: critical | high | medium | low | info | none
--json	off	Output results as JSON instead of a table
--show-secrets	off	Print full token values instead of redacted previews
--ignore TEXT	—	Comma-separated glob patterns to skip (e.g. "*.test.js,fixtures/*")
--max-size INTEGER	10	Maximum file size to scan, in MB
--recursive / --no-recursive	--recursive	Scan subdirectories recursively

Exit codes

Code	Meaning
0	No findings at or above --fail-on severity
1	One or more findings detected at or above --fail-on severity
2	Error (bad argument, unreadable file, etc.)

Other commands

harpocrates version          # Print version
harpocrates serve            # Start the REST API server (requires harpocrates[api])
harpocrates --help           # Full command list

---

How it scans

Harpocrates runs a three-phase pipeline on every line of every file:

1. Regex — deterministic patterns for known credential formats (AWS, GitHub, Stripe, private keys, and more). No ML required. High-confidence, zero false positives on well-formed keys.

2. Entropy analysis — Shannon entropy flags high-randomness tokens that don't match any known pattern. Catches credentials stored under ambiguous variable names (my_key, token, secret) that regex scanners miss entirely.

3. ML verification (opt-in via --ml) — a single-stage XGBoost classifier extracts 64 features from the token, its variable name, and the surrounding code context. It learns to distinguish api_secret = "AKIA..." (secret) from commit_sha = "a1b2c..." (Git SHA) without relying on the variable name alone. Inference runs via ONNX Runtime when available, with native XGBoost as fallback.

Ships pre-trained. No user training required.

The ML model is bundled with the package. pip install "harpocrates[ml]" is all you need.

---

MCP Server

Harpocrates ships an MCP (Model Context Protocol) server that exposes scan_text and scan_file to compatible agents via stdio transport.

Install:

pip install "harpocrates[mcp]"   # requires Python 3.10+

Run:

harpocrates-mcp                  # listens on stdio, speaks JSON-RPC

Tools exposed:

Tool	Arguments	Returns
scan_text	text: str, include_token: bool = False, include_contributions: bool = False	List of finding dicts
scan_file	path: str, include_token: bool = False, max_bytes: int|None = None, include_contributions: bool = False	List of finding dicts

Tokens are redacted by default (include_token=False). Pass include_contributions=True to receive TreeSHAP explanations (requires pip install harpocrates[ml]). The MCP process has the same filesystem read scope as the user who launched it.

---

Explainability

Harpocrates ships opt-in TreeSHAP explanations for ML-stage findings via --explain. The default scan path is unchanged — XGBoost is not imported unless --explain is passed.

Install:

pip install "harpocrates[ml]"

CLI:

# Emit JSON with per-feature SHAP contributions (implies --ml)
harpocrates scan ./my_project --explain

# Inspect the top features driving the first finding
harpocrates scan ./my_project --explain | jq '.findings[0].explanation.top_positive'

Output shape:

{
  "findings": [
    {
      "finding": { "type": "ML_CANDIDATE", "severity": "high", "category": "api_token", ... },
      "explanation": {
        "finding_id": "a1b2c3d4e5f6a7b8",
        "category": "api_token",
        "base_log_odds": -1.4,
        "raw_log_odds": 2.8,
        "predicted_probability": 0.943,
        "top_positive": [
          { "name": "var_ngram_secret_score", "index": 12, "value": 0.873, "contribution": 1.85, "direction": "positive" },
          { "name": "token_entropy",          "index": 3,  "value": null,  "contribution": 1.10, "direction": "positive" }
        ],
        "top_negative": [ ... ],
        "contributions": [ ... ]
      }
    }
  ]
}

explanation is null for regex-tier findings (they have no model decision to explain). Token-derived feature values (token_entropy, token_length, etc.) are suppressed to null in the output to prevent token reconstruction.

MCP: pass include_contributions=true to scan_text or scan_file to receive explanations inline.

---

Version history

v0.4.0 — Redesigned 64-feature vector, v0.4 retrain, opt-in TreeSHAP explainability

Feature engineering (Phase 7.0 / 7.0.11):

• Dropped 8 shortcut / collinear features that caused the model to memorise narrow synthetic distributions rather than generalise. Dropped: var_contains_secret (25% importance but a direct mirror of the heuristic layer — label leak), is_known_hash_length (inverted signal on real credentials), cryptographic_score, normalized_entropy (collinear with token_entropy + char counts), line_position_ratio (train/serve skew from synthetic estimation), hex_context_git_keywords, cross_line_entropy, contains_example_keyword (collinear with their sibling features). Net feature count: 65 → 64.
• Added 7 value-shape features that distinguish credentials from file paths, enum constants, host configs, and template placeholders — the four false-positive classes from real-world scans: value_starts_with_slash, value_contains_path_separator, value_ends_with_known_ext, value_is_lowercase_word, value_is_dotted_quad_or_host_literal, value_is_template_syntax, is_hex_with_no_alpha_mix.
• Tightened XGBoost regularisation (max_depth 6→5, n_estimators 500→300, L1/L2 increased) to break reliance on shortcut features.

v0.4 retrain (Phase 7):

• Retrained on ~40k samples (v4 corpus) with the redesigned 64-feature vector.
• New negative-class generators covering the four v0.3.0 FP classes: file-path values, enum constants (lowercase words), host/port configs, template placeholders (${...}, {{...}}, __X__), file-extension values (.pem, .jks, .p12).
• New positive-class generators: APIM-style compound var names (APIM_CLIENT_KEY, APIM_SECRET_KEY), multiline structures (PEM blocks, YAML pipe-block secrets, k8s Secret manifests, .env multikey blocks, JSON arrays of keys, Terraform for_each secret maps) encoded via context_before/context_after.
• 300-sample hand-curated positive holdout fixture (tests/fixtures/positive_holdout_v4.jsonl); golden OOD recall 97.33% (gate: ≥97%).

Opt-in explainability (Phase 8):

• --explain flag on harpocrates scan emits structured JSON with per-feature TreeSHAP contributions. Implies --ml. Default scan path unchanged — xgboost is never imported without --explain.
• include_contributions=True on MCP scan_text/scan_file tools adds an explanation key to each finding dict.
• All float outputs rounded to 3 decimal places. Token-derived feature values (token_entropy, token_length, char_class_count, digit_ratio, special_char_ratio) suppressed to null in JSON output to prevent token reconstruction.
• Booster cached per process (first --explain call ~30ms; subsequent calls ~0.2ms each).
• Hot-path invariant enforced by tests/test_hot_path_no_xai.py (subprocess isolation + AST static analysis).

Severity calibration (v0.3.2, folded into this release train):

• _entropy_severity() stub replaced with a classification-aware helper — entropy/ML findings now surface at MEDIUM or HIGH instead of always INFO.
• Suffix-style var-name lexicon added (*_KEY, *_SECRET_KEY, *_CLIENT_KEY, etc.) so APIM_CLIENT_KEY, APIM_SECRET_KEY classify as api_token at confidence 0.80 → MEDIUM.
• OPENAI_API_KEY_LEGACY regex added to HIGH_SIGNATURES — catches legacy/short OpenAI keys with hyphens (sk-RQMJj8ELDjv7TRc-...) missed by the strict 48-char CRITICAL pattern.

---

v0.3.0 — ML model retrained on 62k samples, comment scanning, violation classification

New ML capabilities (what the model can now detect that it couldn't before):

• Commented-out secrets — all comment styles now scanned: # (Python/YAML/Ruby), // (JS/TS/Go/Java), /* */ (C/CSS/Java), <!-- (HTML), -- (SQL/Lua). A credential commented out of active code is still in your git history forever.
• AI-scaffolded credentials — trained on LLM-generated code patterns where AI assistants hardcode credentials during integration scaffolding (OpenAI, Stripe, Twilio, Terraform, Helm, Dockerfile).
• Runtime generation negatives — the model now correctly ignores secrets.token_hex(32), crypto.randomBytes(32), uuid.uuid4().hex, SecureRandom.getInstanceStrong(). These are runtime generation calls, not stored secrets. v0.2 would flag them.
• Dev/prod swap patterns — KEY = "sk_test_x" # PROD: sk_live_y — the trailing comment leaks the production key.
• Duplicate-token lines — password = "password123", token = "token-prod-abc...". The ML model now handles lines where the variable name appears as a substring of the value, which confused earlier models.
• Env-var fallback leaks — expanded coverage across Django (config(..., default=)), Pydantic Settings, Spring Boot @Value, Rails credentials.dig, Go, Node.js (??), TypeScript/Zod, Helm | default.
• Violation classification — every finding now includes a category field (password, api_token, connection_string, jwt, private_key, crypto_key, oauth_secret, session_token, webhook_url, generic_secret) and a category_reason string explaining which heuristic fired. This tells you what to rotate, not just that something was found.

Model metrics (v5 corpus, 62k training samples):

Set	Recall	Precision	F1	AUC-ROC
Synthetic test (internal)	99.15%	94.61%	0.968	0.9957
Held-out OOD (TruffleHog-validated)	97.32%	87.14%	0.920	0.9823

Threshold low (recall gate): 0.15 — anything above this is flagged for review. Threshold high (precision gate): 0.85 — anything above this is reported as a confirmed secret.

---

v0.2.0 — Three-phase pipeline, MCP server, pre-commit hook

ML capabilities added:

• XGBoost + ONNX Runtime — single-stage classifier with 64 features. Replaces the v0.1 heuristic confidence scores with a calibrated probability (Platt-scaled).
• 64-feature context model — captures token entropy, variable name semantics (var_ngram_secret_score), file type risk (file_is_config, file_extension_risk), surrounding context (context_has_function_def), and value structure (is_hex_with_no_alpha_mix, value_starts_with_slash, value_is_dotted_quad_or_host_literal, value_is_template_syntax).
• Dual-threshold routing — findings are routed to SAFE (below threshold_low), REVIEW (between thresholds), or SECRET (above threshold_high). The review zone is ~5% of lines on average.
• Entropy candidates surfaced — high-entropy tokens under any variable name now reach the ML stage, not just those matching known patterns.
• MCP server — harpocrates-mcp exposes scan_text and scan_file as JSON-RPC tools for agent-to-agent scanning (Claude, Cursor, Codeium, etc.).
• TokenMatch infrastructure — internal span tracking for exact token position within a line, required for the v0.3 feature-set retrain.

Regex signatures added in v0.2:

Credential	Pattern
OpenAI API key	sk-... / sk-proj-...
Anthropic Claude key	sk-ant-api03-...
GCP API key	AIza...
NPM token	npm_...
PyPI token	pypi-...
HashiCorp Vault token	hvs....

---

v0.1.0 — Initial release ("the wedge")

What shipped:

• Three-phase detection pipeline: Regex → Shannon Entropy → ML confidence scoring.
• 10 regex signatures: AWS Access Key ID, GitHub PAT, Slack token, Stripe key, private key PEM header, Slack webhook, Discord webhook, SendGrid key, Twilio SID, Databricks token.
• Pre-commit hook — scans every staged file before commit. Handles binary files and UnicodeDecodeError without crashing.
• CLI: harpocrates scan, --json, --fail-on, --ml-threshold, --show-secrets, --ignore.
• Packaged as py3-none-any.whl — no native extensions, installs on any platform without compilation.

---

Contributing

False negatives are the highest-priority reports. If Harpocrates missed a real secret, open an issue with the false-negative label and include the variable name pattern and secret type. This is the most valuable feedback you can give.

For bugs, feature requests, and false positives, open an issue at github.com/Skipa776/Harpocrates/issues.

---

License

MIT — see LICENSE.