hydra-logger 0.7.0

A dynamic, multi-headed logging system for Python applications

HYDRA-LOGGER

hydra-logger is a modular Python logging library for teams that need configurable logging without coupling application code to fixed transports or formats.

Best fit:

• teams that need structured logging with strict reliability controls
• services that route logs by layer, destination, and policy
• organizations migrating from ad-hoc logging to consistent standards

Overview

Core capabilities:

• Sync, async, and composite logger types
• Layer-based routing with per-layer destinations and levels
• Console, file, network, and null handlers
• Plain text, colored, JSON lines, and structured formats
• Optional extensions (for example security and performance)

Design principles:

• Keep implementation simple and maintainable
• Favor configuration over hardcoded behavior
• Keep module boundaries explicit and extensible

Quick links:

• Environment setup
• Enterprise tutorials
• Configuration reference
• Operations diagnostics
• Release policy & API compatibility
• Testing and quality gates
• Performance and benchmarks

Why Teams Choose Hydra-Logger

Hydra-Logger gives teams production-ready logging with clear defaults, strong reliability controls, and high throughput.

• Built for real systems: sync, async, and composite logger runtimes
• Policy-driven routing: layers, per-destination levels, and typed network destinations
• Safety-first: strict reliability guards, path controls, and extension-based data protection
• Proven performance: benchmarked profiles with drift and reliability checks (nightly_truth)

Latest nightly snapshot (benchmark/results/benchmark_latest.json):

• Reliability status: passed (strict mode enabled)
• Drift status: passed
• Sync throughput: ~53.7k msg/s
• Async throughput: ~44.5k msg/s
• Concurrent throughput: ~283.4k msg/s

Benchmark details:

• benchmark/README.md
• benchmark/results/benchmark_latest_summary.md

Start quickly:

pip install hydra-logger
.hydra_env/bin/python examples/tutorials/python/t01_production_quick_start.py

Examples and tutorials:

• Tutorial tracks: examples/tutorials/README.md
• Full examples catalog: examples/README.md
• Run examples individually with .hydra_env/bin/python <script_path>.
• Most tutorials write under examples/logs/tutorials/ (gitignored); network tutorials may emit JSON artifacts there too.
• Notebooks: open from repo root or set HYDRA_LOGGER_REPO; see examples/tutorials/notebooks/README.md.
• CI: tutorial scripts and layout are covered by tests/examples/ (pytest tests/examples -q).

Install

pip install hydra-logger

Core install includes only required baseline dependencies. Optional extras install dependency bundles for specific integration scenarios (including custom handlers and application-side adapters), for example:

pip install "hydra-logger[network]"
pip install "hydra-logger[perf]"
pip install "hydra-logger[database,cloud,queues]"
pip install "hydra-logger[full]"

Notes about extras:

• network supports built-in typed network destinations (network_http, network_ws, network_socket, network_datagram).
• database, cloud, queues, and system provide optional dependency bundles for advanced/custom integrations.

Development environment:

# Option A (venv)
python3 -m venv .hydra_env
source .hydra_env/bin/activate
python -m pip install --upgrade pip setuptools wheel
python -m pip install -e .[dev]
python -m pip install pyright

# Option B (Conda prefix)
conda env create -p ./.hydra_env -f environment.yml
source "$(conda info --base)/etc/profile.d/conda.sh"
conda activate "$(pwd)/.hydra_env"

Environment maintenance and troubleshooting are documented in docs/ENVIRONMENT_SETUP.md.

Production operator baseline

Recommended starting posture for services where lost or silent-dropped logs matter:

• Enable strict_reliability_mode and set reliability_error_policy to warn or raise (see docs/RELEASE_POLICY.md and enterprise tutorials).
• Prefer performance_profile minimal or balanced on hot paths; keep convenient for diagnostics-heavy workloads (docs/PERFORMANCE.md).
• Treat async_cloud and similar schema-level destinations as integration points: core ships no default sink—wire a custom handler or expect NullHandler with policy-governed diagnostics (docs/modules/config.md, docs/modules/handlers.md).
• For network_ws, default remains simulated; opt into real I/O with use_real_websocket_transport=True on WebSocketHandler (or use_real_websocket_transport on the LogDestination) and the network extra (websockets).
• On shutdown, inspect get_health_status() for handler_close_failures / last_lifecycle_error when using strict reliability settings (docs/OPERATIONS.md).
• Do not rely on regex redaction as DLP—see docs/OPERATIONS.md.

Anti-patterns: assuming PyPI version matches a git checkout without running scripts/release/check_pypi_parity.py --require-match after publish; enabling cloud destinations without adapters; ignoring UserWarning from simulated WebSocket emits.

Quick Start

from hydra_logger import LoggingConfig, LogLayer, LogDestination, create_logger

config = LoggingConfig(
    layers={
        "app": LogLayer(
            destinations=[
                LogDestination(type="console", format="colored", use_colors=True),
                LogDestination(type="file", path="app.log", format="json-lines"),
            ]
        )
    }
)

with create_logger(config, logger_type="sync") as logger:
    logger.info("Application started", layer="app")
    logger.warning("Low memory", layer="app")
    logger.error("Database connection failed", layer="app")

Async variant:

import asyncio
from hydra_logger import create_async_logger


async def main():
    async with create_async_logger("MyAsyncApp") as logger:
        await logger.info("Async logging works")
        await logger.warning("Async warning message")


asyncio.run(main())

Configuration

Format configuration:

config = LoggingConfig(
    layers={
        "app": LogLayer(
            destinations=[
                LogDestination(type="console", format="json", use_colors=True),
                LogDestination(type="file", path="app.log", format="plain-text"),
                LogDestination(type="file", path="app_structured.jsonl", format="json-lines"),
            ]
        )
    }
)

Destination configuration:

config = LoggingConfig(
    layers={
        "api": LogLayer(
            destinations=[
                LogDestination(type="console", format="colored"),
                LogDestination(type="file", path="api.log", format="json-lines"),
            ]
        )
    }
)

Typed network destination configuration (FastAPI-style DX):

config = LoggingConfig(
    layers={
        "webhook": LogLayer(
            destinations=[
                LogDestination(
                    type="network_http",
                    url="https://logs.example.com/ingest",
                    timeout=5.0,
                    retry_count=3,
                    retry_delay=0.5,
                )
            ]
        ),
        "streaming": LogLayer(
            destinations=[
                LogDestination(
                    type="network_ws",
                    url="wss://stream.example.com/events",
                    timeout=10.0,
                    retry_count=5,
                    retry_delay=1.0,
                )
            ]
        ),
    }
)

Network migration guidance:

• Prefer explicit typed destinations: network_http, network_ws, network_socket, network_datagram.
• Legacy network remains transitional and is mapped to network_http when url is provided.
• Update legacy network configs incrementally to typed variants to avoid future deprecation friction.

Network behavior (operational):

• HTTP probe: HTTPHandler runs a separate connectivity probe before use. Default probe uses GET (backward compatible). Set LogDestination.probe_method to HEAD, OPTIONS, or none, or connection_probe=False, if your ingest endpoint must not observe GET side effects.
• WebSocket: WebSocketHandler currently uses a simulated transport (records are not sent over the wire); a UserWarning is emitted once per process on first emit. Treat network_ws as simulation until a real client integration ships.
• Unknown / unsupported destinations resolve to a no-op NullHandler. With reliability_error_policy="warn" or strict reliability, the runtime surfaces this instead of failing silently.
• Composite async direct I/O: CompositeAsyncLogger with use_direct_io=True flushes file writes via a thread pool when an event loop is running, to avoid blocking the loop; sync close() still flushes on-thread.

Extension configuration:

config = LoggingConfig(
    enable_data_protection=True,
    extensions={
        "data_protection": {
            "enabled": True,
            "type": "security",
            "patterns": ["email", "phone", "api_key"],
        }
    }
)

Optional async runtime queue mode (opt-in, default behavior unchanged):

config = LoggingConfig(
    layers={
        "default": LogLayer(destinations=[LogDestination(type="async_file", path="app.jsonl")])
    },
    extensions={
        "async_runtime": {
            "mode": "queue",              # default is task scheduling mode
            "worker_count": 2,            # async queue workers
            "max_queue_size": 20000,      # bounded queue for backpressure
            "overflow_policy": "drop_newest",  # drop_newest | drop_oldest | block_with_timeout
            "put_timeout_seconds": 0.01,  # used when overflow_policy=block_with_timeout
        }
    },
)

Enterprise hardening profile (strict reliability is opt-in and does not change default template behavior):

from hydra_logger.config.defaults import get_enterprise_config

config = get_enterprise_config()
# Highlights:
# - strict_reliability_mode=True
# - reliability_error_policy="warn"
# - enforce_log_path_confinement=True
# - allow_absolute_log_paths=False

Log file location policy:

• hydra_logger does not create log directories on import/install.
• Log directories are created only when file destinations are configured and initialized.
• Default behavior (no base_log_dir) writes to <current working directory>/logs.
• For strict confinement to project-owned paths, set:

config = LoggingConfig(
    base_log_dir="logs",
    enforce_log_path_confinement=True,
    allow_absolute_log_paths=False,
)

Architecture

System flow (high-level):

flowchart TD
  n_app["Application Code"] --> n_call["logger.info message call"]
  n_call --> n_logger_layer["Logger Layer"]
  n_logger_layer --> n_sync["SyncLogger"]
  n_logger_layer --> n_async["AsyncLogger"]
  n_logger_layer --> n_composite["CompositeLogger"]
  n_logger_layer --> n_composite_async["CompositeAsyncLogger"]
  n_sync --> n_layer_manager["Layer Manager"]
  n_async --> n_layer_manager
  n_composite --> n_layer_manager
  n_composite_async --> n_layer_manager
  n_layer_manager --> n_handlers["Handler System"]
  n_handlers --> n_console["Console"]
  n_handlers --> n_file["File"]
  n_handlers --> n_network["Network"]

Detailed architecture and workflow documentation:

• docs/ARCHITECTURE.md
• docs/WORKFLOW_ARCHITECTURE.md
• docs/modules/README.md

Operations

Quality and validation commands:

# Environment preflight
.hydra_env/bin/python scripts/dev/check_env_health.py --strict

# Test gate
.hydra_env/bin/python -m pytest -q

# Run all examples
.hydra_env/bin/python examples/run_all_examples.py

# Tutorial guardrails (assets, runner, notebook factory contract)
.hydra_env/bin/python -m pytest tests/examples -q

# Performance benchmark
.hydra_env/bin/python benchmark/performance_benchmark.py

# Runtime guard (forbid blocking runtime calls in hydra_logger)
.hydra_env/bin/python -m pytest tests/quality/test_runtime_blocking_calls.py -q

Enterprise tutorial tracks:

.hydra_env/bin/python examples/tutorials/python/t01_production_quick_start.py
.hydra_env/bin/python examples/tutorials/python/t03_layers_customization.py
.hydra_env/bin/python examples/tutorials/python/t04_extensions_plugins.py
.hydra_env/bin/python examples/tutorials/python/t07_operational_playbook.py
.hydra_env/bin/python examples/tutorials/python/t10_enterprise_profile_config.py
.hydra_env/bin/python examples/tutorials/python/t11_enterprise_policy_layers.py
.hydra_env/bin/python examples/tutorials/python/t12_network_http_typed_destination.py
.hydra_env/bin/python examples/tutorials/python/t13_network_ws_resilient_typed_destination.py
.hydra_env/bin/python examples/tutorials/python/t14_network_local_http_simulation.py

Documentation

Start Here (Users)

• Environment setup and troubleshooting
• Enterprise tutorials index
• Examples catalog
• Configuration reference

Design and Runtime

• Architecture overview
• Runtime workflow architecture
• Module map and ownership

Operate and Validate

• Operations diagnostics
• Testing strategy and quality gates
• Performance guidance
• Benchmark usage guide

Troubleshooting

• Environment health checks
• Runtime diagnostics and triage
• Performance regression troubleshooting

Maintainers and Release

• Release policy (semver, classifiers, defaults)
• Release checklist
• Audit sign-off matrix · PyPI publish + parity verify
• Agent automation workflow
• Changelog

Internal planning and audit trackers are maintained under docs/plans/, docs/audit/, and docs/archive/ for repository governance and historical traceability.

Contributing

• Keep changes focused and maintain backward compatibility for public APIs
• Add or update tests in tests/ for behavior changes
• Update docs when behavior or public interfaces change
• Run pre-commit and .hydra_env/bin/python -m pytest -q before opening a PR
• Run release preflight before tagging/publishing: .hydra_env/bin/python scripts/release/preflight.py
• Follow docs/RELEASE_CHECKLIST.md for release evidence and final gate order

License

MIT. See LICENSE.