redact-secret-adapters

Keep secrets out of Python logs and OpenTelemetry traces. Tokens, API keys and passwords are replaced before a record reaches a handler or a span reaches an exporter.

Built on Redact Secret, which does the detection.

Install

pip install redact-secret redact-secret-adapters           # logging
pip install redact-secret "redact-secret-adapters[otel]"   # + OpenTelemetry

Needs Python 3.10 or later.

Quick start

logging

import logging
from redact_secret_adapters.logging_filter import RedactSecretFilter

handler = logging.StreamHandler()
handler.addFilter(RedactSecretFilter())  # on the handler, not the logger
logging.getLogger().addHandler(handler)

token = "ghp_SYNTHETICREVOKED00000000000000000000"  # synthetic
logging.warning("deploy with token %s", token)
# deploy with token <SECRET_1>

Your formatters, extra configuration and exception logging keep working unchanged.

Where to attach it

A logging.Filter runs only where it is attached. This is the one thing to get right: add the filter to every handler that writes somewhere. To prove it for your own handlers, run the placement recipe: it logs a synthetic token through real handlers and includes an unfiltered handler as a negative control.

import logging
from redact_secret_adapters.logging_filter import RedactSecretFilter

console = logging.StreamHandler()
audit = logging.FileHandler("audit.log")

redact = RedactSecretFilter()  # no per-record state: one instance can be shared
for handler in (console, audit):
    handler.addFilter(redact)  # every emitting handler, not the logger
    logging.getLogger().addHandler(handler)

This is not global automatic protection. A handler added anywhere else (by a library, by logging.basicConfig, by a child logger of your own) is unprotected until it, too, carries the filter.

Handlers run in the order they were added and the filter mutates the record in place, so a filtered handler also protects every handler after it, and an unfiltered handler that runs before it emits plaintext. Do not rely on order: filter each one.

For a QueueHandler/QueueListener pair, attach the filter to the QueueHandler. It runs in the emitting thread, so only masked records cross the queue:

handler = logging.handlers.QueueHandler(records)
handler.addFilter(RedactSecretFilter())
listener = logging.handlers.QueueListener(records, logging.StreamHandler())

python/tests/test_logging_placement.py asserts each supported placement and, as synthetic negative controls, that plaintext really does escape each wrong one.

What is covered

After formatting the message the filter clears args, so a downstream formatter cannot rebuild the original. A record with no finding is formatted byte-for-byte as it would be without the filter. Masking an already-masked record again is a no-op, so two filtered handlers on one record are safe.

What a named extra can be:

The last row fails closed on purpose. The filter cannot know what a %(ctx)s format or a JSON formatter's default=str would print for an arbitrary object, so it never hands one over unscanned. Convert such a value to a dict or a str yourself before logging it if you want it kept.

Nothing raises into your logging call. A container whose read raises (a __getitem__, keys() or slice that raises) becomes [REDACTED:ERROR] for that entry, or for the whole container when it cannot be listed at all. A message that cannot be formatted (a bad % format, a raising __str__) becomes [REDACTED:ERROR].

Filter options

RedactSecretFilter(
    extra_fields=["user"],
    on_outcome=observe,
    policy=policy,
    limits={"max_depth": 4},
    operation_limits={"max_leaves": 1000},
    ruleset=ruleset,
)

OpenTelemetry ([otel] extra)

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from redact_secret_adapters.otel import create_redacting_span_processor

provider = TracerProvider()
provider.add_span_processor(create_redacting_span_processor(BatchSpanProcessor(otlp_exporter)))

Wrap the processor you already have. It takes policy, limits and on_outcome.

OpenInference and GenAI semantic-convention attributes are covered without hardcoding either convention's attribute list.

A span that cannot be redacted is dropped, not exported. opentelemetry-sdk has no public way to change a span before export, so the adapter writes the private fields behind the read-only accessors (_name, _status, _attributes and its backing _dict) and reads each write back through the public accessor. If an SDK release moves one of those fields, the span is dropped rather than exported unredacted, and a RuntimeWarning is issued once per processor. See redact_secret_adapters.otel for details.

Supported range: opentelemetry-sdk>=1.16.0,<2. CI runs tests/test_otel_host.py, a real TracerProvider/exporter round trip, at both ends of that range on every run.

Masking callbacks (Langfuse and similar)

from redact_secret_adapters.mask_secrets import mask_secrets

langfuse = Langfuse(mask=mask_secrets)

mask_secrets(data=...) returns a masked copy. mask_secrets_with(scan, data) is the same walk over an injected scanner.

PII detection is opt-in

Credential detection needs no init step: the native extension loads on import redact_secret. PII detection does. It is opt-in, process-wide and one-shot, and the application turns it on:

import redact_secret

redact_secret.initialize(pii=["pii:global"])  # before the first record or span

Or let the factory do it. Pass pii= to the live constructor or factory and it initializes the core and verifies the selection is active before it returns, so there is no silent window:

import logging
from redact_secret_adapters.logging_filter import RedactSecretFilter
from redact_secret_adapters.otel import create_redacting_span_processor

handler = logging.StreamHandler()
handler.addFilter(RedactSecretFilter(pii=["pii:global"]))  # adapter first

provider.add_span_processor(
    create_redacting_span_processor(exporter_processor, pii=["pii:global"])  # same contract
)

Logging and OpenTelemetry share one implementation. An equivalent selection that is already active is accepted, so application-first plus pii= works. Anything else raises redact_secret_adapters.CoreActivationError at construction, before any record or span: .code is PII_ACTIVATION_NOT_ACTIVE (a different selection won the race), PII_ACTIVATION_UNAVAILABLE (the core does not offer a selector) or PII_ACTIVATION_UNSUPPORTED (a core too old to report an activation). The message is fixed and never carries a selector, an input or the core's own text; other initialization failures propagate unchanged. pii= cannot be combined with an injected scan_and_redact, which never imports or initializes the real core. pii=[] asks for PII off explicitly, which is also one-shot.

Without pii=, placement is the whole rule. Handlers are attached and tracer providers built at import time, so a module imported earlier can emit before the initialize line runs. Those records are scanned with PII off and report nothing: no exception, no warning, and no counter that tells them apart from a record that genuinely held nothing. Omitting pii= leaves activation to the application and keeps that window, pinned as a known limitation in python/tests/test_pii_activation.py. Enable PII first, or pass pii=.

The selection is one-shot: a later different one raises redact_secret.PiiActivationConflictError, and an empty selection is a different selection, not a neutral one.

Activation is not masking. Under the core's default policy, PII types are confidence-gated rather than always redacted: a High-confidence finding redacts, while Medium and Low resolve to warn, and a warn finding leaves the text alone. Enabling PII therefore still lets lower-confidence PII reach a handler or an exporter as plaintext. Pass your own policy mapping those findings to redact if you need them masked. The counters make it observable: a record whose values.findings is non-zero while values.redacted stays at zero is exactly this case.

Counting what happened

on_outcome reports one summary per unit: one logging record, or one span. It is observational: increment your own counters from it. Nothing here creates a logger, a handler, an exporter or a network client.

from redact_secret_adapters.logging_filter import RedactSecretFilter


def observe(outcome):  # LogRecordOutcome(level=..., values=ValueCounts(...))
    metrics.increment("log.records", level=outcome.level)
    metrics.increment("log.redacted_values", outcome.values.redacted)


handler.addFilter(RedactSecretFilter(on_outcome=observe))

from redact_secret_adapters.otel import create_redacting_span_processor


def observe(outcome):  # SpanOutcome(values=ValueCounts(...), dropped=False)
    if outcome.dropped:
        metrics.increment("span.dropped_unredactable")


provider.add_span_processor(create_redacting_span_processor(next_processor, on_outcome=observe))

A ValueCounts is six non-negative integers and nothing else. There is no field for a value, a record attribute, a key, an offset or an exception message:

• The units follow placement: a record through two filtered handlers is two passes and reports twice, which is what a per-handler count means. The second pass finds nothing left to redact.
• SpanOutcome.dropped is this processor's own decision (a masked value would not write back). It is not a claim that an exporter succeeded, nor that a span was sampled out.
• Both observers run after the record or span is fully masked, so neither can turn a protected one into an unprotected one. Anything they raise is swallowed, never read, and never re-raised.
• The re-entrancy guard is thread-local: an observer that logs or traces does not recurse, and one thread never suppresses another's outcome.

Fail-closed markers

When something goes wrong, a fixed marker is written instead of the original text. What each one means and how to correct it: troubleshooting.

DEFAULT_LIMITS:

max_nodes counts every value the walk visits: containers and leaves alike, but not dict keys. A value reached by more than one path counts once per path. The walk does not track values it has already visited, so a structure built from shared lists or dicts is walked once per path, and without this budget its cost would grow exponentially. Past max_nodes, every value, a number included, becomes [REDACTED:LIMIT_EXCEEDED].

A limits override that is not a usable bound (None, NaN, a negative number, a bool, or not a number at all) falls back to that key's default, instead of raising or disabling the bound. float("inf") is a valid bound and means none.

mask_secrets_with, mask_log_value_with and extra_fields share one walker, so the extras table under What is covered holds for all three: any object that is not a string, number, boolean, None, dict, list, tuple or exception becomes [REDACTED:ERROR]. The TypeScript walker instead serializes an object the way JSON.stringify would; Python has no single serialization to mirror.

Dict keys and attribute names are not scanned on their own. The walker masks values only; a string directly under a key is also scanned in its key-context view {"<key>":"<value>"} (the core decides; a finding outside the value blocks the value and keeps the key), the way the JavaScript walker does. Every dict key, and every attribute name copied from an exception's __dict__, reaches the handler unchanged, and so does the name of an extra_fields attribute. The OpenTelemetry processor likewise leaves every span, event, and link attribute key as it is. Do not put a secret in a key or an attribute name.

Core scan options

RedactSecretFilter, RedactingSpanProcessorWith, create_redacting_span_processor, redact_attributes_with, mask_secrets_with and mask_log_value_with pass three more core options through, unchanged, on every scan, beside policy:

Policy precedence. There is one policy and the adapter never combines two: your policy replaces the core's built-in policy for every finding, including those a ruleset detector adds. Under the core's default policy a ruleset finding is Medium confidence and only warns, which leaves the text alone, so supply a policy that returns redact or block for it. A block finding still replaces the whole leaf, and a warn still leaves the text alone, on top of what the policy returned.

The options are validated and snapshotted once, when the filter or processor is built (resolve_scan_config; per call for mask_secrets_with): a scan_limits mapping is copied and a bytearray ruleset becomes bytes, so mutating yours afterwards changes nothing; a malformed option is a TypeError with a fixed message. With the real core (the filter built with no scanner argument, and create_redacting_span_processor) a scan_limits mapping becomes redact_secret.WholeInputLimits; give mask_secrets_with and the other *_with functions a WholeInputLimits when the scanner you inject is the real core. A leaf the core refuses is [REDACTED:ERROR], never the input or the exception's message. A keyed leaf is also scanned in its key-context view, so under a byte ceiling its usable size is max_input_bytes minus the key and punctuation.

An installed core that cannot honor a requested option is rejected, not ignored: the live constructors check redact_secret.VERSION against SCAN_OPTION_CORE_FLOORS (every option is available from the declared floor, 0.1.0-beta.6) and probe with one scan of the empty text, so an older core or a ruleset or limits it refuses raises CoreOptionsError with a fixed message and code (CORE_OPTION_UNSUPPORTED / CORE_OPTION_REJECTED), the option names, and the core's own core_code (INVALID_RULESET, ...), never a value or the core's message. A core that omits every new option is untouched. All of this is whole-input; the core has no ruleset for an incremental session.

Aggregate operation budget

limits bound one walk and max_string_length one string. A log record is masked as a message, an exception text, a stack and several extra fields, and a span has many attributes, events and links, so many individually valid fields could multiply the total work. An operation is one filter() call, one span ending or one mask_secrets_with call, and it owns one redact_secret_adapters.budget.OperationBudget that every pass and field of it shares; each call gets a fresh one, so two threads never share a budget.

Occurrences are counted where a value is visited (a shared reference reached by two paths counts twice); actual calls where work is done. Bytes are UTF-8. The outcome counters keep their meaning (scanned counts leaves), and the per-walk limits still apply. Exhaustion is sticky and deterministic: past a bound every string not yet inspected becomes [REDACTED:LIMIT_EXCEEDED] unscanned (counted as limited), mapping keys past it are dropped, and the record or span is still delivered, never with text the budget did not allow to be inspected. It is a work counter, not a wall-clock timeout: it is checked between scans, and a scan_and_redact call or a host callback that never returns is not interrupted. The same options are accepted by mask_secrets_with, mask_log_value_with, RedactingSpanProcessorWith, create_redacting_span_processor and redact_attributes_with as operation_limits.

Development

From the repository root:

pip install -e "./python[otel,test]"
pytest

The fixture tests read the same JSON files in the repository's fixtures/ directory as the TypeScript suite; that shared file is what keeps the two languages equivalent. More: CONTRIBUTING.md. Changes are listed in this package's CHANGELOG.md.

License

MIT