safestream-redactor

Detect and redact PII and credentials in text files of any size, with constant memory. The file is streamed in chunks, so a 100 GB log uses the same memory as a 1 KB one. Bytes outside a redacted span, line endings included, are left exactly as they were.

Requirements

• Python 3.10 or newer, on Linux, macOS or Windows.
• No third-party runtime dependencies (tomli only on Python 3.10). Optional extras are listed below.

Installation

pip install safestream-redactor

Only need the command-line tool? Install it into an isolated environment with pipx:

pipx install safestream-redactor

Check the install:

safestream --version                # prints e.g. "safestream 0.4.0"
python -m safestream_redactor --version

Optional extras

An optional compiled (Rust) core gives the same output as the pure-Python pattern and labeled-identifier tiers, 3-12x faster on the project's evaluation corpora; enable it with Redactor(backend="native"). Its wheel is not on PyPI yet, so build it from source (needs a Rust toolchain):

pip install "safestream-redactor-native @ git+https://github.com/MounishSenisetty/SafeStream-Redactor#subdirectory=native"

Without it, backend="native" falls back to pure Python with a warning.

Upgrade or remove:

pip install --upgrade safestream-redactor
pip uninstall safestream-redactor

Command-line usage

The package installs one command, safestream, with six subcommands:

Run safestream <command> --help for the full option list.

redact

# redact everything detectable
safestream redact input.txt -o output.txt

# only some entity types, with a custom replacement
safestream redact input.txt -o output.txt --types email,ssn --replace "***"

# keep the last 4 characters visible
safestream redact cards.csv -o masked.csv --mode mask --keep-last 4

# consistent keyed tokens, e.g. <EMAIL_75e802c7c314f5a2> (same input -> same token)
safestream redact input.txt -o output.txt --mode pseudonymize --hmac-key "$SECRET"

# per-type replacement text, plus your own words and regexes
safestream redact input.txt -o output.txt --replace-type email='<EMAIL>' \
    --custom-word ProjectX --custom-regex 'INT-\d+'

# stdin to stdout
cat app.log | safestream redact - -o -

# a whole directory, in parallel; print counts and write a JSON report
safestream redact logs/ -o redacted/ --workers 4 --stats --report run.json

Example:

$ printf 'contact bob@corp.io, ssn: 123-45-6789\n' | safestream redact - -o -
contact [REDACTED], ssn: [REDACTED]

--report FILE writes counts and sizes only, never the matched text.

detect

safestream detect app.log                     # human-readable list (no secret text)
safestream detect app.log --json              # one JSON object per detection
safestream detect src/*.py --sarif > out.sarif  # SARIF 2.1.0
safestream detect . --fail-on-detect          # exit status 1 if anything is found
safestream detect app.log --show-text         # also print the matched text (local only)

$ safestream detect app.log
app.log:3:7: email  conf=0.95
app.log:9:12: ssn  conf=1.00
-- 2 detection(s) (matched text hidden; --show-text reveals it)

init-config

safestream init-config -o policy.toml
safestream redact input.txt -o output.txt --config policy.toml

Flags given on the command line override the file. Unknown keys are rejected.

[detection]
# types = ["email", "ssn", "credit_card"]   # omit for all types
min_confidence = 0.5
use_ner = false
use_entropy = true

[redaction]
mode = "replace"                            # replace | mask | pseudonymize
replacement = "[REDACTED]"
mask_keep_last = 4

[redaction.replacements]
# email = "<EMAIL>"

[custom]
# words = ["ProjectX"]
# patterns = ['\\binternal-[a-z]+\\b']

serve

safestream serve                     # http://127.0.0.1:8000 (loopback only)
safestream serve --port 9000 --config policy.toml --max-body 1048576

curl -s -X POST http://127.0.0.1:8000/redact \
  -H 'Content-Type: application/json' \
  -d '{"text": "email bob@corp.io"}'
# {"redacted": "email [REDACTED]", "total_detections": 1, "detections_by_type": {"email": 1}}

Endpoints: GET /health, GET /version, GET /detectors, POST /redact, POST /scan. Request bodies are never logged.

mcp-serve

Exposes the tools scan, redact, explain_detection, detectors and health to an MCP client over stdio. Register it in your client's configuration as:

{ "command": "safestream", "args": ["mcp-serve"] }

ui

Opens an interactive playground in your browser (requires the ui extra):

pip install 'safestream-redactor[ui]'
safestream ui                          # http://127.0.0.1:8501

Paste text or upload a .txt, .log, .json or .csv file, up to 2 GB by default (--max-upload-mb). Uploads over 5 MiB are redacted by the parallel streaming engine and previewed, and the redacted file can be saved to a folder. Streamlit holds an upload in memory (about twice its size) while it is selected. Detect highlights findings by severity, with a table of types, confidence and line:col locations. Redact shows the original and sanitized text side by side (replace, mask, or keyed hash) and lets you download the result. Inspect replays the streaming engine one sliding window at a time. It shows the context, emitted, and carried bands, and how an entity that a chunk boundary would cut is kept whole. The page listens on 127.0.0.1 only, with Streamlit usage statistics off and the offline network guard on. A banner confirms these settings, or warns when they are missing.

Common options (redact and detect)

Redaction options (redact only): --mode {replace,mask,pseudonymize}, --replace TEXT, --replace-type TYPE=TEXT, --keep-last N, --mask-char C, --hmac-key KEY. Mask mode never leaves more than half of a value visible, so a short value is never shown in full.

Detection options (detect only): --json, --sarif, --fail-on-detect, and --show-text / --reveal-matches (also print the matched text, which is hidden by default so detect output is safe to log; for local triage only). detect accepts files and directories; directories are walked recursively, skipping symlinks, .git/, and binary files.

While redact or detect runs, an in-process guard blocks outbound (non-loopback) network connections by default. On Linux, --offline-hard adds a kernel-enforced block on top.

Exit status

Entity types

Python API

from safestream_redactor import EntityType, RedactionPolicy, Redactor

redactor = Redactor()
redactor.redact("email bob@corp.io, ssn: 123-45-6789")
# 'email [REDACTED], ssn: [REDACTED]'

# inspect without redacting
[(d.entity_type.value, d.text) for d in redactor.detect("mail bob@corp.io from 10.0.0.1")]
# [('email', 'bob@corp.io'), ('ipv4', '10.0.0.1')]

# choose types and how they are rewritten
Redactor(types=["email"]).redact("bob@corp.io 123-45-6789")
# '[REDACTED] 123-45-6789'
policy = RedactionPolicy(replacements={EntityType.EMAIL: "<EMAIL>"})
Redactor(policy=policy).redact("write to bob@corp.io")
# 'write to <EMAIL>'
Redactor(policy=RedactionPolicy(mode="mask", mask_keep_last=4)).redact("card 4111 1111 1111 1111")
# 'card ***************1111'

# your own words and patterns
Redactor(custom_words=["ProjectX"], custom_patterns=[r"INT-\d+"]).redact("ProjectX ticket INT-42")
# '[REDACTED] ticket [REDACTED]'

# file to file with constant memory; the output is written atomically
stats = redactor.redact_file("huge.log", "huge.redacted.log")
stats.summary()   # '2 detection(s): email=1, ssn=1'

# split one large file across processes (same output as a single process)
redactor.redact_file("huge.log", "huge.redacted.log", workers=8)

# any iterable of text chunks (a socket, a generator, a file object)
with open("app.log", newline="") as source:
    for piece in redactor.redact_stream(source):
        print(piece, end="")

Adding a detector from another package

A package can add a detection tier by declaring an entry point. Plugins are opt-in: enable them with Redactor(load_plugins=True), --enable-plugins, or [plugins] enabled = true (a warning names any installed plugin that was not loaded):

[project.entry-points."safestream_redactor.detectors"]
my_detector = "my_package.detectors:MyDetector"

A detector is any object with a name attribute and a detect(text) -> list[Detection] method.

Limitations

Structured data (e-mails, cards, SSNs, keys, tokens) is detected with high precision. Names and street addresses in free prose are only detected with the ner or onnx extra, and even then on a best-effort basis. Review redacted output before sharing it where a miss would be costly.

Links

• Source and full documentation: https://github.com/MounishSenisetty/SafeStream-Redactor
• Issues: https://github.com/MounishSenisetty/SafeStream-Redactor/issues
• Changelog: https://github.com/MounishSenisetty/SafeStream-Redactor/blob/main/CHANGELOG.md

License

MIT