scan-for-secrets 0.3

Scan for secrets in files you plan to share

scan-for-secrets

Scan for secrets in files you plan to share

Installation

Install this tool using pip:

pip install scan-for-secrets

Or uv:

uv tool install scan-for-secrets

Or use without installing via uvx:

uvx scan-for-secrets --help

Usage

This tool helps scan all of the text files in a directory (ignoring binary files) to see if they include specified secret strings. For example, run this if you want to publish the logs from a coding agent session after first confirming no secrets from environment variables are exposed in those logs.

Basic usage looks like this:

scan-for-secrets $OPENAI_API_KEY $ANTHROPIC_API_KEY

This will scan text files in the current folder and all sub-folders looking for the values that were passed as positional arguments, including common escaping schemes that might mean a direct string match misses them.

To scan for a secret that can be accessed using another command, use $(command) syntax:

scan-for-secrets "$(llm keys get openai)"

Add -d/--directory to specify a different directory to scan. This can be passed multiple times:

scan-for-secrets $OPENAI_API_KEY -d ~/my-project
scan-for-secrets $OPENAI_API_KEY -d ~/project-a -d ~/project-b

Use -f/--file to scan specific files instead of (or in addition to) directories. This can also be passed multiple times. Missing files are silently ignored.

scan-for-secrets $OPENAI_API_KEY -f output.log -f debug.json
scan-for-secrets $OPENAI_API_KEY -d ~/project -f ~/extra-log.txt

If neither -d nor -f is provided, the current directory is scanned.

You can also pipe a list of newline-separated secrets to the tool:

cat secrets.txt | scan-for-secrets

This can be combined with secrets passed as positional arguments.

Add -v/--verbose to see which directories are being scanned (output goes to stderr). In verbose mode, any matches found are repeated at the end of the output so they aren't lost in the directory listing:

scan-for-secrets $OPENAI_API_KEY -v

Redacting secrets

Use -r/--redact to replace found secrets with REDACTED directly in the scanned files. The tool will show all matches first, then ask for confirmation before rewriting anything:

scan-for-secrets $OPENAI_API_KEY -r

Example interaction:

logs/2024-03-15.jsonl:42: sk-a... (literal)
logs/2024-03-15.jsonl:108: sk-a... (json)

Replace 2 occurrences in 1 file with REDACTED?
Proceed? [y/N]: y
Replaced 2 occurrences.

All escaped variants of the secret (JSON, URL-encoded, etc.) are replaced as well. If no secrets are found, no prompt is shown. If you decline the prompt, the tool exits with code 1 (same as finding secrets without --redact).

Note: when using --redact, secrets cannot be piped via stdin since stdin is reserved for the confirmation prompt. Pass secrets as arguments or use a config file instead.

Output

If no secrets are found, the tool will terminate with an exit code 0 and output nothing. If secrets are found it will return an exit code 1 and list the files, line numbers and the first few characters of each secret that was spotted.

Example output:

logs/2024-03-15.jsonl:42: sk-a... (literal)
logs/2024-03-15.jsonl:108: sk-a... (json)
config/debug.html:7: ghp_... (html)

Configuration file

If you run scan-for-secrets without any extra arguments or piped data the command will look for a default configuration file to tell it what to scan for instead.

This file lives at ~/.scan-for-secrets.conf.sh and contains commands that will be executed to retrieve secrets. Each line should be a shell command that outputs a single secret to stdout (or a blank line or a comment).

# API keys
echo $OPENAI_API_KEY
echo $ANTHROPIC_API_KEY

# AWS (using xargs to strip whitespace)
awk -F= '/aws_secret_access_key/{print $2}' ~/.aws/credentials | xargs

# 1Password
op read "op://Vault/API Key/password"

# LLM keys
llm keys get gemini

Blank lines and lines starting with # are ignored. By default the file is executed with sh. Add a shebang line (e.g. #!/bin/bash or #!/usr/bin/env python3) to use a different interpreter.

With a configuration file setup you can run scan-for-secrets like this:

cd agent-logs/
scan-for-secrets

Or this:

scan-for-secrets -d agent-logs

You can also pass a path to a configuration file using the -c/--config option:

scan-for-secrets -c scan.sh

Unlike the default configuration behavior, this -c option will be combined with any piped data or additional positional arguments.

Using this as a Python library

This package can also be used as a Python library. Add scan-for-secrets as a dependency and use it like this:

from scan_for_secrets import scan_directory

result = scan_directory("./logs", ["sk-abc123...", "ghp_secret..."])

if result.has_secrets:
    for match in result.matches:
        print(f"{match.file_path}:{match.line_number}: {match.secret_hint} ({match.encoding})")

API reference

scan_directory(directory: str | Path, secrets: list[str]) -> ScanResult

Recursively scans all text files in directory for the given secrets, checking both literal matches and common escaped variants (JSON, URL percent-encoding, HTML entities, backslash-doubled and Unicode escapes). Returns a ScanResult with all matches collected.

• directory: Root directory to scan. Can be a string path or a pathlib.Path.
• secrets: List of secret strings to search for. Empty strings are ignored.

Binary files (detected by null bytes in the first 8192 bytes) are skipped. The following directories are also skipped: .git, .hg, .svn, node_modules, __pycache__, .venv, venv.

scan_directory_iter(directory, secrets, on_enter_directory=None) -> Iterator[Match]

def scan_directory_iter(
    directory: str | Path,
    secrets: list[str],
    on_enter_directory: Callable[[str], None] | None = None,
) -> Iterator[Match]:

Streaming version of scan_directory — yields Match objects as they are found instead of collecting them. Useful for large directory trees where you want to display results immediately.

The optional on_enter_directory callback is called with the relative path of each directory as it is entered.

from scan_for_secrets import scan_directory_iter

for match in scan_directory_iter("./logs", ["sk-abc123...", "ghp_secret..."]):
    print(f"{match.file_path}:{match.line_number}: {match.secret_hint} ({match.encoding})")

scan_file(file_path: str | Path, secrets: list[str]) -> ScanResult

Scan a single file for secrets. Returns a ScanResult with files_scanned always set to 1. The file_path field on each match will be the file's basename.

from scan_for_secrets import scan_file

result = scan_file("/path/to/output.log", ["sk-abc123..."])
if result.has_secrets:
    for match in result.matches:
        print(f"{match.file_path}:{match.line_number}: {match.secret_hint}")

redact_file(file_path: str | Path, secrets: list[str], replacement: str = "REDACTED") -> int

Replace all occurrences of the given secrets (including escaped variants) in a single file. Returns the number of replacements made. The file is only rewritten if at least one replacement occurs.

from scan_for_secrets import redact_file

count = redact_file("/path/to/output.log", ["sk-abc123..."])
print(f"Replaced {count} occurrences")

scan_file_iter(file_path: str | Path, secrets: list[str]) -> Iterator[Match]

Streaming version of scan_file — yields Match objects as they are found. The file_path field on each match will be the file's basename.

from scan_for_secrets import scan_file_iter

for match in scan_file_iter("/path/to/output.log", ["sk-abc123..."]):
    print(f"{match.file_path}:{match.line_number}: {match.secret_hint} ({match.encoding})")

ScanResult

@dataclass
class ScanResult:
    matches: list[Match]  # All matches found across all files
    files_scanned: int    # Number of text files checked

    @property
    def has_secrets(self) -> bool:
        """True if any matches were found."""

Match

@dataclass
class Match:
    file_path: str     # Path relative to the scanned directory
    line_number: int   # 1-based line number where the match was found
    secret_hint: str   # First 4 characters of the original secret + "..."
    encoding: str      # How the secret was encoded: "literal", "json", "url",
                       # "html", "backslash-doubled", or "unicode-escape"

Escaping schemes

In addition to literal string matching, scan-for-secrets checks for these escaped forms of each secret:

• JSON (json) — Characters are escaped as they would appear inside a JSON string: \", \\, \/, \n, \t, and \uXXXX for non-ASCII characters. Catches secrets embedded in JSON files, API responses, and log output from JSON-based tools.
• URL percent-encoding (url) — Every non-alphanumeric character is replaced with %XX hex encoding (e.g. = becomes %3D, & becomes %26). Catches secrets in URLs, query strings, and form data.
• HTML entities (html) — & < > " are replaced with named entities (&, <, >, "), and non-ASCII characters become numeric references like Ã. Catches secrets embedded in HTML pages and XML documents.
• Backslash-doubled (backslash-doubled) — Every \ is replaced with \\. Catches secrets in configuration files, YAML, TOML, and other formats that escape backslashes.
• Unicode escape (unicode-escape) — Non-ASCII characters are replaced with Python-style escape sequences like \xe9 or \u00e9. Catches secrets in source code and debug output.

If an encoding produces the same string as the literal secret (for example, URL-encoding a plain alphanumeric string), that redundant variant is skipped.

Development

To contribute to this tool, first checkout the code. Then run the tests:

cd scan-for-secrets
uv run pytest

To run the development version of the command itself:

uv run scan-for-secrets --help