secretHider

Fast secret redaction, log processing, string and JSON tools: one C++20 core, thin bindings for Python and Node.js.

• Remove secrets from logs in plain text, logfmt and JSON lines: AWS/GitHub/Slack/Stripe/Google keys, JWTs, PEM private keys, Bearer tokens, URL credentials, password=... fields, and optional PII (email, phone, card, SSN, IBAN, IPv4).
• Tell it what is secret with plain lists: known values, field names and formats (or one secrets.txt file), on top of the built-in patterns.
• Choose what happens: delete the whole field (drop_field), mask (full, typed, keep first/last N, preserve length, partial email/phone), stable salted hash, realistic fake data via Faker (Python faker, JS @faker-js/faker), or your own function. Partial masks never reveal more than 25% of a secret.
• Stream it: chunk-safe redaction for files, pipes and sockets (output equals one-shot output at every split point), a secrethider-redact command line, and in-place/.gz file helpers.
• Fast: about 600 MiB/s per core on clean logs, roughly 38x a hand-written Python re pipeline and comparable to V8's RegExp for a few simple patterns in Node (numbers). Built on simdjson, simdutf, RE2.
• Also: json_get (JSON Pointer), json_minify, json_valid, UTF-8 helpers.

secrethider-redact --in-place --drop-fields --secrets-file secrets.txt app.log   # a=1 password=x b=2  ->  a=1 b=2
secrethider-redact --pii --mask fake app.jsonl --json-lines                       # fake emails/phones instead of [REDACTED]

import secrethider
from secrethider import Mask, RedactorBuilder

r = (RedactorBuilder()
     .enable("pii")
     .secret_values(["my-known-secret"])       # also: secret_keys([...]), secret_formats([...])
     .mask(Mask.keep_prefix(4))                # or Mask.drop_field(), secrethider.fake_mask(), ...
     .build())

r.redact("login password=hunter2hunter2hunter2 key=AKIAIOSFODNN7EXAMPLE")
# 'login password=hunt**** key=AKIA****'
# (short secrets fall back to a full mask: 4 revealed characters of a 10-character password would exceed 25%)

# logging, files, streams
from secrethider.logging import RedactingFormatter
handler.setFormatter(RedactingFormatter(r, "%(levelname)s %(message)s"))
secrethider.redact_file(open("in.log"), open("out.log", "w"), r)

const fk = require('@noobforal/secrethider');
const r = new fk.RedactorBuilder().mask(fk.Mask.keepPrefix(4)).build();
process.stdin.pipe(fk.createRedactStream(r)).pipe(process.stdout);

Install

pip install secrethider            # Python 3.9+; add `faker` for realistic fake-data masks
npm install @noobforal/secrethider # Node.js 18+ (prebuilt binaries for Linux/macOS/Windows x64 and macOS arm64)

The npm package is scoped because npm rejects the unscoped name secrethider as too similar to an existing package.

Layout

core/              C++20 library (no Python/Node headers), tests, benchmarks
bindings/python/   nanobind module + `secrethider` package (built via scikit-build-core, root pyproject.toml)
bindings/node/     Node-API addon (node-addon-api + cmake-js)
cmake/             shared CMake modules (dependencies, warnings)
docs/              architecture: interfaces, log processing, rules and masking, builds, performance, security

Documentation

Start with the guide to removing secrets from logs, or docs/README.md for everything: interfaces, log processing pipeline, rules and masking, build and platforms, performance, security model.

Develop

make fix     # format + auto-fix lint: C++ (clang-format), Python (ruff), JS/TS (prettier)
make test    # build, run all tests (C++, Python, Node), then the benchmarks
make help    # everything else: lint, check (tests only), bench, bench-full, qa (leak checks + long fuzz), clean

make finds Visual Studio's C++ tools on Windows by itself. Without make, the underlying steps are:

# C++ core: tests + benchmarks (Windows: use the dev-msvc preset from a VS developer prompt)
cmake --preset dev && cmake --build --preset dev && ctest --preset dev
./build/dev/core/secrethider_bench                  # secrethider_bench.exe on Windows

# Python
python -m venv .venv && . .venv/bin/activate    # Windows: .venv\Scripts\activate
pip install . pytest faker && pytest

# Node
cd bindings/node && npm install && npm run build && npm test

-DSECRETHIDER_WITH_RE2=OFF skips RE2/abseil for a much faster build; custom regex() rules are then unavailable.

Cross-platform builds

• Python wheels: cibuildwheel (Linux manylinux/musllinux, macOS x86_64 + arm64, Windows), stable-ABI (abi3) wheels, via .github/workflows/release.yml.
• Node: Node-API is ABI-stable, so one binary per OS/arch; CI collects prebuilds/<platform>-<arch>/secrethider.node.
• SIMD is selected at run time by simdjson/simdutf, so no -march=native and one binary runs fast everywhere.

Limits

Redaction is defence in depth, not a guarantee: it finds known formats, sensitive key names and values you register. See the security model.

License

MIT, see LICENSE.