py-hpl-logger 0.1.0

A high-performance, buffered, non-blocking logger for Python, implemented in Rust.

High-Performance Python Logger (in Rust)

A high-performance, buffered, non-blocking logger for Python, with the core logic implemented in Rust for maximum speed and efficiency.

This logger is designed for high-throughput applications where standard Python logging would become a bottleneck. It sends logs on a dedicated background thread, ensuring your application's main threads are never blocked by I/O.

Features

• Non-Blocking: Log calls return instantly.
• Batching: Logs are sent to destinations in efficient batches (Elastic bulk API).
• Resilient: In-memory buffer with retries for when destinations are temporarily unavailable.
• Multiple Outputs: Configure logging to stdout, files, and Elasticsearch simultaneously.
• Python logging Integration: Includes a logging.Handler to seamlessly integrate with the standard library.

Requirements

• Python 3.9+
• pip version 21 or newer.

This package uses modern Python packaging standards (PEP 517). Older versions of pip may not be able to install it correctly. You can upgrade pip with the following command:

pip install --upgrade pip

Pre-compiled wheels for officially Supported Platforms

• Linux: x86_64 (manylinux compatible)
• macOS: arm64 (Apple Silicon)

Installation

pip install py-hpl-logger

Quick Start

1. Configure your logger with ElasticConfig if needed:

   ELASTIC_HOST=localhost
   ELASTIC_PORT=9200
   ELASTIC_USERNAME=elastic
   ELASTIC_PASSWORD=changeme
   ELASTIC_INDEX=my-python-logs
   

   Configure using code:

   elastic_config = ElasticConfig(
       host="localhost",
       port=9200,
       index="my-app",
       username="elastic",
       password="changeme"
   )
   

   Configure using .env. You can choose .env search depth using local_only argument:

   elastic_config = ElasticConfig.from_env(local_only=False)
   

2. Use the logger in your Python application:

   import logging
   from py_hpl_logger import LoggerBuilder, ElasticConfig, RustLogHandler
   
   # 1. Build the Rust logger backend once
   elastic_config = ElasticConfig(
       host="localhost",
       port=9200,
       index="my-app",
       username="elastic",
       password="changeme"
   )
   rust_backend = (
       LoggerBuilder()
       .with_stdout(True) # whether to use stdout or not
       .with_file_output("my_app_session")
       .with_elastic_output(elastic_config)
       .with_batch_size(1000) # how many log rows to wait before forced flush
       .with_flush_interval(1.0) # # how many seconds to wait before forced flush
       .build()
   )
   
   # 2. Integrate with Python's standard logging
   handler = RustLogHandler(rust_logger=rust_backend)
   formatter = logging.Formatter("%(threadName)s - %(message)s")
   handler.setFormatter(formatter)
   logger.addHandler(handler)
   
   # 3. Use the standard logging API anywhere!
   log = logging.getLogger(__name__)
   log.info("This log is being handled by Rust!")
   log.error("This is a high-performance error log.")
   
   # 4. For graceful shutdown, flush the logger before exiting
   # (The logger also attempts to flush automatically on exit)
   rust_backend.flush()
   

License

This project is licensed under the MIT License.