elven-logs-interceptor-python

High-performance, production-ready log interceptor for Python with Loki transport, batching, compression, circuit breaker, DLQ, and framework integrations.

Installation

pip install elven-logs-interceptor-python

With all extras:

pip install "elven-logs-interceptor-python[all]"

Quick Start

from logs_interceptor import init, logger

init(
    {
        "appName": "billing-service",
        "interceptConsole": True,
        "transport": {
            "url": "https://loki.example.com/loki/api/v1/push",
            "tenantId": "tenant-a",
            "authToken": "token",
            "compression": "gzip",
        },
    }
)

logger.info("service started", {"port": 3000})

Logging compatibility

The logger accepts both the Elven structured style and common Python logging call patterns. This makes migration safer when legacy code replaces logging.getLogger(__name__) with the Elven proxy.

logger.info("payload created", {"request_id": "req-1"})
logger.info("payload created: %s", payload)
logger.info("payload created: %(id)s", {"id": payload_id})
logger.info("payload created: {}", payload_id)
logger.info("payload created: %s", payload_id, extra={"request_id": "req-1"})
logger.exception("failed to create payload: %s", payload_id)

When intercepting the standard Python logging module, extra={...} is captured under context.extra:

logging.getLogger(__name__).info(
    "payload created",
    extra={"payload": payload},
)

Supported aliases: warning(...), exception(...) and critical(...).

Sensitive fields such as cpf, cnpj, cpfCnpj, documento, numeroDocumento, taxId, apiKey, clientSecret, senha, password, token, authorization, cartao/cartão, chavePix and related credential fields are redacted by default. In messages, the library redacts only the sensitive fragment/value and preserves the rest of the log. In structured context, sensitive keys redact the full value. The key matcher normalizes common snake_case, camelCase, kebab-case, dotted variants, and accents. Set LOGS_FILTER_SANITIZE=false, LOGS_SANITIZE=false, or UO_LOGS_SANITIZE=false only when raw sensitive data is explicitly required and approved.

Environment Variables

The package supports the Python LOGS_* configuration surface aligned with the JS v3 design where applicable.

Required for direct Loki mode:

• LOGS_URL
• LOGS_TENANT
• LOGS_APP_NAME

For collector-first mode, set LOGS_EXPORTER=otlp (or collector) and use OTEL_EXPORTER_OTLP_LOGS_ENDPOINT. If the explicit logs endpoint is missing, the library derives /v1/logs from OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, or OTEL_EXPORTER_OTLP_ENDPOINT. In this mode LOGS_URL, LOKI_URL, LOGS_TENANT and LOGS_TOKEN are not required by the library. LOGS_URL is intentionally ignored in collector mode to avoid accidentally posting OTLP logs to a direct Loki /loki/api/v1/push endpoint. Put tenant or auth headers in OTEL_EXPORTER_OTLP_HEADERS/LOGS_OTLP_HEADERS if your collector gateway requires them.

Core:

• LOGS_TOKEN
• LOGS_APP_VERSION
• LOGS_ENVIRONMENT

Transport:

• LOGS_EXPORTER (loki|otlp, aliases: collector, otel)
• OTEL_EXPORTER_OTLP_LOGS_ENDPOINT (collector logs endpoint, e.g. http://collector:4318/v1/logs)
• LOGS_OTLP_HEADERS (comma-separated headers, e.g. x-tenant=team-a)
• LOGS_COMPRESSION (none|gzip|brotli|snappy)
• LOGS_COMPRESSION_LEVEL
• LOGS_COMPRESSION_THRESHOLD
• LOGS_USE_WORKERS
• LOGS_MAX_WORKERS
• LOGS_WORKER_TIMEOUT
• LOGS_CONNECTION_POOLING
• LOGS_MAX_SOCKETS
• LOGS_ENABLE_STRUCTURED_METADATA (false by default; enable only when Loki allows limits_config.allow_structured_metadata)
• LOGS_TIMEOUT
• LOGS_MAX_RETRIES
• LOGS_RETRY_DELAY

Buffer:

• LOGS_BUFFER_MAX_SIZE
• LOGS_BUFFER_FLUSH_INTERVAL
• LOGS_BUFFER_MAX_MEMORY_MB
• LOGS_BUFFER_MAX_AGE
• LOGS_BUFFER_AUTO_FLUSH

Filter:

• LOGS_FILTER_LEVELS
• LOGS_FILTER_SAMPLING_RATE
• LOGS_FILTER_SANITIZE (aliases: LOGS_SANITIZE, UO_LOGS_SANITIZE)
• LOGS_FILTER_MAX_MESSAGE_LENGTH
• LOGS_FILTER_EXCLUDE_LOGGER_PREFIXES (comma-separated)

Circuit Breaker:

• LOGS_CIRCUIT_BREAKER_ENABLED
• LOGS_CIRCUIT_BREAKER_FAILURE_THRESHOLD
• LOGS_CIRCUIT_BREAKER_RESET_TIMEOUT
• LOGS_CIRCUIT_BREAKER_HALF_OPEN_REQUESTS

DLQ:

• LOGS_DLQ_ENABLED
• LOGS_DLQ_TYPE (memory|file)
• LOGS_DLQ_MAX_SIZE
• LOGS_DLQ_MAX_RETRIES
• LOGS_DLQ_BASE_PATH

Runtime:

• LOGS_MAX_CONCURRENT_FLUSHES
• LOGS_INTERCEPT_CONSOLE
• LOGS_PRESERVE_ORIGINAL_CONSOLE
• LOGS_ENABLE_METRICS
• LOGS_ENABLE_HEALTH_CHECK
• LOGS_DEBUG
• LOGS_SILENT_ERRORS
• LOGS_ENABLED
• LOGS_AUTO_INIT
• LOGS_ENABLE_EXPERIMENTAL_PROTOBUF (optional, enables experimental snappy/protobuf transport path)

Labels:

• Prefix LOGS_LABEL_* (example: LOGS_LABEL_SERVICE=billing)

Public API

• init(config)
• get_logger()
• is_initialized()
• destroy()
• adestroy()
• logger proxy with debug/info/warn/warning/error/exception/fatal/critical/log/track_event/flush/aflush/get_metrics/get_health/destroy/adestroy/with_context/with_context_async

Python import remains:

import logs_interceptor

Integrations

• Python logging (LoggingHandler)
• FastAPI / Starlette (FastAPIMiddleware)
• Django (DjangoMiddleware)
• Flask (FlaskExtension)
• Celery (CelerySignals)
• structlog (StructlogProcessor)
• loguru (LoguruSink)

Auto Init

Set LOGS_AUTO_INIT=true and import the package.

Preload mode:

python -m logs_interceptor.preload

Development

pip install -e ".[dev]"
ruff check .
mypy src
pytest

Deploy / Publish

Local publish script:

./scripts/publish.sh --repository testpypi --dry-run
./scripts/publish.sh --repository testpypi
./scripts/publish.sh --repository pypi

Token environment variables used by default:

• TEST_PYPI_API_TOKEN for --repository testpypi
• PYPI_API_TOKEN for --repository pypi

Optional script flags:

• --skip-checks (skip lint/type/test)
• --skip-build (skip build/twine check)
• --token-env CUSTOM_VAR (custom token env var name)
• --no-skip-existing (upload fails if version already exists)

Makefile shortcuts:

make qa
make publish-dry-run
make publish-testpypi
make publish-pypi

GitHub Actions publish workflow:

• File: .github/workflows/publish.yml
• Trigger: manual (workflow_dispatch)
• Inputs: repository = testpypi | pypi
• Required secrets:
  • TEST_PYPI_API_TOKEN
  • PYPI_API_TOKEN

License

MIT