filter-json-log 0.7.1

Safe JSON serialization for logs

filter-json-log

A configurable Python utility to serialize JSON-like data for logs without leaking credentials.

filter-json-log redacts secret-like fields, sanitizes URLs through filter-url, scans emitted strings through a shared sanitizer pipeline, and caps rendered output size so logs stay safe and readable.

Key Features

• JSON-safe logging output: serializes dict, list, tuple, set, scalars and raw JSON strings into valid JSON text suitable for logs.
• Systematic text sanitization: JSON keys, string values, URL components, headers, query strings, form bodies, environment-like data, command strings and rendered log messages all pass through the same sanitizer pipeline.
• URL-aware filtering: URL-like fields are preserved as URLs while userinfo, query values, fragments, path segments and nested URL/query values are redacted where needed.
• Policy modes: choose light, normal, paranoid or lockdown depending on how much readability you are willing to trade for safety.
• Configurable rules: replace or extend default secret markers, URL markers, provider-token regexes and language/domain marker packs.
• Language marker packs: built-in packs include technical English, French, German, Ukrainian and Russian markers.
• Logging integration: includes a logging.Filter subclass that sanitizes extra, positional/mapping arguments and already-rendered messages.
• Output budget control: defaults to max_len=4096 and max_nls=8; when the output cannot fit safely, visible excess is replaced with "[...]".
• Strict raw JSON mode: with raw_str=True, strings must parse as JSON; parse failures render as "[JSON parsing error]" instead of being silently treated as normal strings.

Installation

pip install filter-json-log

filter-json-log is designed to use filter-url for URL sanitization. Package metadata should install it as a dependency; if you are using the module directly from source, install it explicitly:

pip install filter-url

Quick Start

The quickest way to use the library is the standalone filter_json_log() function.

from filter_json_log import filter_json_log

payload = {
    "user": "alice",
    "password": "correct horse battery staple",
    "url": "https://user:[e-mail]/login?token=abc-123",
}

safe = filter_json_log(payload)
print(safe)

Output:

{
  "user": "alice",
  "password": "[REDACTED]",
  "url": "https://[REDACTED]@example.com/login?token=[REDACTED]"
}

Use one_line=True for compact log messages:

safe = filter_json_log(payload, one_line=True)
print(safe)

Output:

{"user":"alice","password":"[REDACTED]","url":"https://[REDACTED]@example.com/login?token=[REDACTED]"}

Usage & Examples

Basic Filtering: Standalone Function

filter_json_log() is convenient for one-off calls. It creates a configured FilterJSONLog instance internally and immediately renders the supplied data.

from filter_json_log import filter_json_log

payload = {
    "request_id": "req-123",
    "headers": {
        "Authorization": "Bearer very-secret-token",
    },
    "redirect_url": "https://client.example/cb#access_token=abc&id_token=jwt",
}

print(filter_json_log(payload, one_line=True))

Output:

{"request_id":"req-123","headers":{"Authorization":"[REDACTED]"},"redirect_url":"https://client.example/cb#access_token=[REDACTED]&id_token=[REDACTED]"}

Raw JSON Strings

By default, a Python string is treated as a JSON string value, not as a JSON document. If you want to parse it as JSON, pass raw_str=True.

from filter_json_log import filter_json_log

raw = '{"password":"secret","url":"https://x.test/?token=abc"}'
print(filter_json_log(raw, raw_str=True, one_line=True))

Output:

{"password":"[REDACTED]","url":"https://x.test/?token=[REDACTED]"}

If raw_str=True is used and the string is not valid JSON, the result is explicit:

print(filter_json_log("not json", raw_str=True))

Output:

"[JSON parsing error]"

This is intentional: when callers promise JSON, malformed input should be visible in the log without leaking the original string.

Advanced: Reuse FilterJSONLog for Performance

When processing many payloads with the same rules, instantiate FilterJSONLog once. This compiles marker rules, regexes and URL filters only once.

from filter_json_log import FilterJSONLog

json_filter = FilterJSONLog(
    mode="normal",
    one_line=True,
    max_len=4096,
    max_nls=8,
)

payloads = [
    {"url": "https://api.example/data?api_key=k1"},
    {"headers": "Authorization: Bearer k2"},
    {"message": "password k3"},
]

safe_payloads = [json_filter.filter(payload) for payload in payloads]

Policy Modes

mode controls how aggressively text is sanitized.

Mode	Intended use	Behavior
light	Maximum readability, legacy systems	Redacts secret-like keys, URL-like fields and strong provider tokens. Free text is touched less aggressively.
normal	Recommended default	Every emitted text atom is sanitized. Marker + payload patterns, URLs, headers, query/form/env/cmd contexts and encoded views are checked.
paranoid	Security-first logs	Uses deeper decoding and stronger marker matching; trades more false positives for fewer leaks.
lockdown	Allowlist-style safety	Unknown string values are redacted unless allowed. Use when low-entropy secrets may appear without markers.

Examples:

from filter_json_log import filter_json_log

payload = {"message": "password LEAK"}

print(filter_json_log(payload, mode="light", one_line=True))
# {"message":"password LEAK"}

print(filter_json_log(payload, mode="normal", one_line=True))
# {"message":"password [REDACTED]"}

Use mode="lockdown" when unknown strings must not leave the process unless explicitly allowed:

payload = {"status": "ok", "note": "blue"}

print(filter_json_log(
    payload,
    mode="lockdown",
    allow_keys={"status"},
    one_line=True,
))
# {"status":"ok","note":"[REDACTED]"}

Output Length and Newline Limits

By default, output is capped at:

max_len = 4096
max_nls = 8

If a value or container cannot be rendered safely within the budget, the visible excess is replaced with "[...]".

from filter_json_log import filter_json_log

payload = {"items": list(range(1000))}
print(filter_json_log(payload, max_len=80, one_line=True))

Output shape:

{"items":[0,1,2,3,4,5,6,7,8,9,"[...]"]}

The exact cut point is budget-dependent. The output remains valid JSON whenever possible.

If max_len is too small, it is normalized to the minimum length needed to render the truncation marker JSON string, currently "[...]".

Disable output caps completely with:

filter_json_log(payload, forced_infinite=True)

Only use forced_infinite=True when you know the payload size is safe for your log system.

URL Fields

URL-like keys have priority over secret-like keys. The value is preserved as a URL but sensitive parts are redacted.

payload = {
    "signed_url": "https://user:[e-mail]/file?X-Amz-Signature=abc",
}

print(filter_json_log(payload, one_line=True))

Output:

{"signed_url":"https://[REDACTED]@example.com/file?X-Amz-Signature=[REDACTED]"}

URL sanitization also checks fragments, nested URL/query values, path segments, path params, protocol-relative URLs and schemeless userinfo-like forms in URL contexts.

Structured Strings: Headers, Query, Form, Env, Cookies and Commands

When a key implies a structured text context, the value is parsed with that context and then routed through the shared sanitizer.

payload = {
    "query": "access_token=abc&next=https%3A%2F%2Fx.test%2F%3Ftoken%3Ddef",
    "headers": "Authorization: Bearer secret\r\n X-Trace: visible",
    "env": "PASSWORD=secret\nDEBUG=true",
    "cmd": "curl -u user:pass https://example.test",
}

print(filter_json_log(payload, one_line=True))

Output shape:

{"query":"access_token=[REDACTED]&next=https%3A%2F%2Fx.test%2F%3Ftoken%3D%5BREDACTED%5D","headers":"Authorization: [REDACTED]\r\n X-Trace: visible","env":"PASSWORD=[REDACTED]\nDEBUG=true","cmd":"curl -u [REDACTED] https://example.test"}

Pair Objects and Pair Arrays

The sanitizer recognizes common key/value shapes used by headers, params, env vars and config dumps.

payload = {
    "headers": [
        {"name": "Authorization", "value": "Bearer abc"},
        ["X-Api-Key", "def"],
    ],
    "env": [
        ["PASSWORD", "secret", "comment"],
    ],
}

print(filter_json_log(payload, one_line=True))

Output shape:

{"headers":[{"name":"Authorization","value":"[REDACTED]"},["X-Api-Key","[REDACTED]"]],"env":[["PASSWORD","[REDACTED]","[REDACTED]"]]}

Language and Domain Marker Packs

Built-in marker packs include:

TECHNICAL_EN_MARKER_PACK
FRENCH_MARKER_PACK
GERMAN_MARKER_PACK
UKRAINIAN_MARKER_PACK
RUSSIAN_MARKER_PACK

You can provide your own MarkerPack for another language or an internal domain vocabulary.

from filter_json_log import MarkerPack, filter_json_log

COMPANY_MARKER_PACK = MarkerPack(
    name="company",
    secret_markers=frozenset({"tenant secret", "internal token"}),
    url_markers=frozenset({"callback endpoint"}),
)

payload = {"message": "tenant secret abc"}

print(filter_json_log(
    payload,
    extra_marker_packs=[COMPANY_MARKER_PACK],
    one_line=True,
))

Output:

{"message":"tenant secret [REDACTED]"}

Custom Secret and URL Rules

You can replace or extend the default markers and regexes.

from filter_json_log import FilterJSONLog

json_filter = FilterJSONLog(
    extra_bad_keys={"tenant_secret"},
    extra_url_keys={"callback_endpoint"},
    extra_strong_embedded_secret_re=[
        r"(?<![A-Za-z0-9])corp_[A-Za-z0-9]{24,}(?![A-Za-z0-9])",
    ],
)

The extra_* parameters extend defaults. The non-extra parameters, such as bad_keys=..., replace the corresponding defaults.

Integration with Python's logging Module

JSONLogFilter is a logging.Filter subclass. It can sanitize:

• selected LogRecord attributes, usually supplied through extra={...};
• mapping-style logging args, for example logger.info("password=%(password)s", {...});
• positional args with simple key inference, for example logger.info("password=%s", secret);
• already-rendered message text, including f-string and .format() messages.

import logging
import sys

from filter_json_log import JSONLogFilter

logger = logging.getLogger("my_app")
logger.setLevel(logging.INFO)
logger.handlers.clear()

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(logging.Formatter("%(levelname)s: %(message)s"))
handler.addFilter(JSONLogFilter(
    mode="normal",
    one_line=True,
    attrs=("payload", "url", "headers"),
))
logger.addHandler(handler)

logger.info(
    "request failed",
    extra={
        "payload": {
            "password": "secret",
            "url": "https://x.test/?token=abc",
        }
    },
)

logger.info("password=%s", "secret")
logger.info("Authorization: Bearer secret")

Output:

INFO: request failed | JSON={"password":"[REDACTED]","url":"https://x.test/?token=[REDACTED]"}
INFO: password=[REDACTED]
INFO: Authorization: [REDACTED]

Logging Filter Options

JSONLogFilter(
    attrs=("json", "payload", "data", "body", "request", "response", "url"),
    fmt=" | JSON={filtered_json}",
    raw_str=False,
    fallback=True,
    append_from_fallback=False,
    replace_attrs=True,
    sanitize_msg=True,
    infer_arg_keys_from_msg=True,
    mode="normal",
)

Notes:

• attrs controls which LogRecord attributes are inspected.
• fmt controls the suffix appended when a selected JSON payload is found.
• sanitize_msg=True sanitizes already-rendered messages. Keep it enabled unless another logging layer already guarantees safety.
• infer_arg_keys_from_msg=True handles common forms like password=%s and Authorization: %s.

Corner Cases & Considerations

This Library Produces Log Text, Not Reusable Payloads

The output is intended for logs, monitoring and debugging. It is valid JSON text in normal operation, but it is not intended to be deserialized and used as a business payload. Redaction markers such as "[REDACTED]" and "[...]" are deliberately human-readable.

Redaction vs Truncation

"[REDACTED]" means the library intentionally hid sensitive data.

"[...]" means the output was truncated because of max_len or max_nls.

These markers have different meanings and should not be collapsed into one value.

False Positives Are Mode-Dependent

light prioritizes readability. normal is the recommended default. paranoid intentionally prefers some false positives over leaks. lockdown redacts unknown strings unless explicitly allowed.

No Sanitizer Can Guess a Secret with No Signal

If a password is the low-entropy string "blue", appears under a neutral key such as "note", and has no marker, URL/query/header/form/cmd context, token shape, encoding clue or entropy signal, no sanitizer can reliably identify it. Use mode="lockdown" with allow_keys for that policy.

Performance

For high-volume logging, instantiate FilterJSONLog once and reuse it. The standalone filter_json_log() function is convenient, but it rebuilds configuration on every call.

Logging Filter Precedence

Passing structured data through extra={"payload": ...} is preferred. Fallback scanning of args and rendered messages is useful, but it costs more CPU and can produce more false positives in aggressive modes.

API Reference

filter_json_log(data, raw_str=False, **kwargs)

Standalone function for one-off JSON log sanitization.

Common parameters passed through **kwargs:

• mode: "light", "normal", "paranoid", or "lockdown". Default: "normal".
• paranoid: backward-compatible alias. paranoid=True means mode="paranoid".
• max_len: maximum output length. Default: 4096.
• max_nls: maximum number of newline characters. Default: 8.
• one_line: render compact JSON without indentation. Default: False.
• forced_infinite: disable length and newline truncation. Default: False.
• redacted: marker for hidden secrets. Default: "[REDACTED]".
• truncated: marker for truncated output. Default: "[...]".
• ensure_ascii: passed to JSON string rendering. Default: False.
• sort_keys: render mapping keys sorted by JSON key text. Default: False.
• raw_str: when True, data must be a JSON string and will be parsed with json.loads().

FilterJSONLog(...)

Reusable class that stores compiled sanitizer configuration.

Important constructor parameters:

• mode, paranoid, max_len, max_nls, one_line, forced_infinite, redacted, truncated, indent, ensure_ascii, sort_keys.
• marker_packs: replace default marker packs.
• extra_marker_packs: add language/domain marker packs to defaults.
• bad_keys, extra_bad_keys: replace or extend secret markers.
• url_keys, extra_url_keys: replace or extend URL markers.
• bad_keys_re, extra_bad_keys_re: replace or extend secret-key regex rules.
• url_keys_re, extra_url_keys_re: replace or extend URL-key regex rules.
• strong_whole_secret_re, extra_strong_whole_secret_re: replace or extend whole-value provider-token regexes.
• strong_embedded_secret_re, extra_strong_embedded_secret_re: replace or extend provider-token regexes found inside strings.
• filter_url_bad_keys, extra_filter_url_bad_keys, filter_url_bad_keys_re, extra_filter_url_bad_keys_re, filter_url_bad_path_re: URL sanitizer configuration.
• paranoid_entropy, entropy_min_length, entropy_threshold: optional entropy-based redaction controls.
• allow_keys: keys that may keep unknown strings in lockdown mode.

Useful methods:

json_filter.filter(data, raw_str=False) -> str
json_filter.filter_text(text, kind="text", key=None) -> str
json_filter.filter_value_for_key(value, key) -> str

MarkerPack(...)

Language/domain pack for semantic markers.

MarkerPack(
    name="custom",
    secret_markers=frozenset({"password", "token"}),
    url_markers=frozenset({"url", "callback"}),
    query_markers=frozenset({"query", "params"}),
    header_markers=frozenset({"headers"}),
    form_markers=frozenset({"form", "body"}),
    env_markers=frozenset({"env", "config"}),
    cookie_markers=frozenset({"cookies"}),
    command_markers=frozenset({"cmd", "argv"}),
)

JSONLogFilter(...)

logging.Filter subclass.

Constructor parameters:

• attrs: LogRecord attributes to inspect. Default includes json, payload, data, body, request, response, params, query, headers, env, cmd, argv, url.
• fmt: suffix appended when selected JSON is found. Default: " | JSON={filtered_json}".
• json_filter_instance: optional preconfigured FilterJSONLog instance.
• raw_str: treat selected string values as raw JSON documents.
• fallback: inspect logging args. Default: True.
• append_from_fallback: append fallback-filtered JSON using fmt. Default: False.
• replace_attrs: replace selected attributes with safe JSON strings. Default: True.
• sanitize_msg: sanitize rendered log message. Default: True.
• infer_arg_keys_from_msg: infer simple key contexts for positional args. Default: True.
• **filter_kwargs: passed to FilterJSONLog if json_filter_instance is not supplied.

License

This project is licensed under the MIT License.