py-hpl-logger 0.2.4

A high-performance, buffered, non-blocking logger for Python, implemented in Rust.

High-Performance Python Logger (in Rust)

A high-performance, buffered, non-blocking logger for Python, with the core logic implemented in Rust for maximum speed and efficiency.

This logger is designed for high-throughput applications where standard Python logging would become a bottleneck. It sends logs on a dedicated background thread, ensuring your application's main threads are never blocked by I/O.

Features

• Non-Blocking: Log calls return instantly.
• Batching: Logs are sent to destinations in efficient batches (Elastic bulk API).
• Resilient: In-memory buffer with retries for when destinations are temporarily unavailable.
• Multiple Outputs: Configure logging to stdout, files, and Elasticsearch simultaneously.
• Python logging Integration: Includes a logging.Handler to seamlessly integrate with the standard library.

Requirements

• Python 3.9+
• pip version 21 or newer.

This package uses modern Python packaging standards (PEP 517). Older versions of pip may not be able to install it correctly. You can upgrade pip with the following command:

pip install --upgrade pip

Pre-compiled wheels for officially Supported Platforms

• Linux: x86_64 (manylinux compatible)
• macOS: arm64 (Apple Silicon, Intel)
• Windows: amd64

Installation

pip install py-hpl-logger

Quick Start

1(Optional). Configure your logger with ElasticConfig if needed:

ELASTIC_HOST=localhost
ELASTIC_PORT=9200
ELASTIC_USERNAME=elastic
ELASTIC_PASSWORD=changeme
ELASTIC_INDEX=my-python-logs

Configure using code:

elastic_config = ElasticConfig(
    host="localhost",
    port=9200,
    index="my-app",
    username="elastic",
    password="changeme"
)

Configure using .env. You can choose .env search depth using local_only argument:

elastic_config = ElasticConfig.from_env(local_only=False)

2. Use the logger in your Python application:

   import logging
   from py_hpl_logger import LoggerBuilder, ElasticConfig, RustLogHandler
   
   # 1. Build the Rust logger backend once
   elastic_config = ElasticConfig(
       host="localhost",
       port=9200,
       index="my-app",
       username="elastic",
       password="changeme"
   )
   # default buffer params:
   # channel_size: 4096
   # batch_size: 256
   # `channel_size` sets maximum inner buffer size. If log amount exceeds this argument, over limiting logs are dropped to avoid memory leaks
   # `batch_size` is an argument used to bulk push amount of logs specified
   rust_backend = (
       LoggerBuilder()
       .with_stdout(True) # whether to use stdout or not
       .with_file_output("my_app_session")
       .with_elastic_output(elastic_config)
       .with_batch_size(1000) # how many log rows to wait before forced flush
       .with_flush_interval(1.0) # # how many seconds to wait before forced flush
       .build()
   )
   
   # 2. Integrate with Python's standard logging
   handler = RustLogHandler(rust_logger=rust_backend)
   formatter = logging.Formatter("%(threadName)s - %(message)s")
   handler.setFormatter(formatter)
   logger.addHandler(handler)
   
   # 3. Use the standard logging API anywhere!
   log = logging.getLogger(__name__)
   log.info("This log is being handled by Rust!")
   log.error("This is a high-performance error log.")
   
   # 4. For graceful shutdown, flush the logger before exiting
   # (The logger also attempts to flush automatically on exit)
   rust_backend.flush()
   

Benchmark

Measurements completed under python3.11 on Apple M2 Max using remote Elastic Search instance with 2 Gb memory limit. The logger version used is py-hpl-logger==0.2.1.

20.000 messages were successfully written in a total time of 2.0091 seconds

    --- Running Benchmark for: High-Performance Rust Logger ---
    Logging 20,000 messages across 10 threads...
    Time taken for High-Performance Rust Logger: 0.1128 seconds
    Throughput: 177,282.63 logs/second

    Flushing Rust logger buffer...
    Flush took an additional 1.8963 seconds.

    --- Benchmark Results ---
    The application code was blocked for 0.1128 seconds with the Rust logger.
    Throughput: 177,282.63 logs/second
    Logs written: 20,000
    Time taken for flush: 1.8963 seconds
    Total time: 2.0091 seconds

Changelog

[0.1.5] - 2025-11-04

• Added 'with_channel_size' constructor to LoggerBuilder

[0.1.6] - 2025-11-04

• Added win_amd64 support
• Added macos_x86_64 support
• Added benchmark results

[0.1.9] - 2025-11-06

• Added 'with_base_log' constructor to LoggerBuilder to add a base prefix '[<event.timestamp>] <event.level>:'
• Fixed potential malloc problems in stdout buffer
• Improved buffer offload with async elastic/file write execution

[0.2.0] - 2025-11-07

• Fixed macos binaries imports

[0.2.1] - 2025-11-08

• Added log file rotation. Now 5 files of 10Mb are set for rotation as a default setting. File rotation can be disabled with 'without_file_rotation' builder
• File rotation can be configured with 'with_file_rotation' builder. It accepts max_file_size_bytes: int, max_backup_files: int as arguments where max_file_size_bytes is the maximum size of each file in bytes and max_backup_files is the maximum number of files to keep
• Added graceful file writing at the cost of an 8% log throughput decrease

[0.2.2] - 2025-11-10

• Fixed stdout only config not producing logs

[0.2.3] - 2025-12-17

• Added protocol argument for ElasticConfig. Now Elastic can be configured to use https hosts. (http -> https redirects not supported for safety reasons)

   elastic_config = ElasticConfig(
       host="localhost",
       port=9200,
       index="my-app",
       username="elastic",
       password="changeme"
       protocol="https", # you can skip protocol setup, default fallback is "http"
   )

[0.2.4] - 2025-12-18

• Added gzip bulk payload compression
• Added WARNING and DEBUG levels

License

This project is licensed under the MIT License.