Litewave Logger

This module provides a centralized and consistent logging solution for Litewave services. It ensures that a request-id is maintained across all services, including HTTP requests and Celery tasks, allowing for easy tracing of requests as they propagate through the system.

Features

• Centralized Logging: A single module to configure and manage logging across all services.
• JSON Logging: All logs are formatted as JSON for easy parsing and integration with log aggregation systems.
• Request ID Propagation: Automatically injects a request-id into all log messages.
• FastAPI Integration: Middleware for FastAPI to handle request-id for incoming HTTP requests and adds it to response headers.
• Flask Integration: Middleware for Flask to handle request-id for incoming HTTP requests and adds it to response headers.
• Celery Integration: Signal handlers to propagate the request-id to Celery tasks automatically.
• Requests Library Patching: Automatically injects the request-id into outgoing HTTP requests made with the requests library.
• HTTPX Patching: Automatically injects the request-id into outgoing HTTP requests made with httpx (sync and async clients).
• Endpoint Exclusion: Configure endpoints that should not be logged (e.g., health checks, metrics).

Installation

1. Add the litewave_logger directory to your Python project.
2. Ensure that the dependencies listed in the main requirements.txt (fastapi, celery, requests) are installed.

Usage

To use the litewave_logger in your service, follow these steps:

1. Initialize the logger: In your main application file (e.g., api.py), import and call the setup_logging function. This should be done as early as possible.

   from litewave_logger import setup_logging
   
   # Optionally exclude endpoints from logging (e.g., health checks, metrics)
   setup_logging(excluded_endpoints=['/health', '/metrics'])
   

2. Add the FastAPI middleware: If your service is a FastAPI application, add the RequestIdMiddleware to your FastAPI app.

   from fastapi import FastAPI
   from litewave_logger.middleware import RequestIdMiddleware
   
   app = FastAPI()
   app.add_middleware(RequestIdMiddleware)
   

   Or for Flask applications: If your service is a Flask application, use the FlaskRequestIdMiddleware.

   from flask import Flask
   from litewave_logger.flask_middleware import FlaskRequestIdMiddleware
   
   app = Flask(__name__)
   FlaskRequestIdMiddleware(app)
   

3. Patch the requests library: To ensure the request-id is propagated to other services via sync HTTP calls, patch the requests library.

   from litewave_logger.requests import patch_requests
   
   patch_requests()
   

4. Patch httpx: To propagate the request-id via async HTTP calls (e.g. httpx.AsyncClient), patch httpx.

   from litewave_logger.httpx_patch import patch_httpx
   
   patch_httpx()
   

5. Connect Celery signals: If your service uses Celery, you need to import the Celery module to ensure the signal handlers are registered. The signal handlers are automatically connected via decorators, so you don't need to call them directly.

   # Just import the module - signal handlers are automatically registered
   import litewave_logger.celery
   

FastAPI Example

Here's a complete example of how to integrate the litewave_logger into a FastAPI application:

from fastapi import FastAPI
from litewave_logger import setup_logging
from litewave_logger.middleware import RequestIdMiddleware
from litewave_logger.requests import patch_requests
from litewave_logger.httpx_patch import patch_httpx

# Import Celery module to register signal handlers (if using Celery)
import litewave_logger.celery

# 1. Initialize logging (optionally exclude endpoints)
setup_logging(excluded_endpoints=['/health', '/metrics'])

# 2. Patch outgoing HTTP libraries
patch_requests()
patch_httpx()

app = FastAPI()

# 3. Add RequestIdMiddleware
app.add_middleware(RequestIdMiddleware)

# Your application code here...

Flask Example

Here's a complete example of how to integrate the litewave_logger into a Flask application:

from flask import Flask
from litewave_logger import setup_logging
from litewave_logger.flask_middleware import FlaskRequestIdMiddleware
from litewave_logger.requests import patch_requests

# Import Celery module to register signal handlers (if using Celery)
import litewave_logger.celery

# 1. Initialize logging (optionally exclude endpoints)
setup_logging(excluded_endpoints=['/health', '/metrics'])

# 2. Patch requests library
patch_requests()

app = Flask(__name__)

# 3. Add FlaskRequestIdMiddleware
FlaskRequestIdMiddleware(app)

@app.route('/')
def hello():
    return 'Hello, World!'

# Your application code here...

The Flask middleware also supports the application factory pattern:

from flask import Flask
from litewave_logger import setup_logging
from litewave_logger.flask_middleware import FlaskRequestIdMiddleware

middleware = FlaskRequestIdMiddleware()

def create_app():
    setup_logging(excluded_endpoints=['/health'])
    app = Flask(__name__)
    middleware.init_app(app)
    return app

How It Works

Request ID Flow

1. Incoming HTTP Request: The RequestIdMiddleware checks for an x-request-id header. If present, it uses that value; otherwise, it generates a new UUID.
2. Context Variable: The request ID is stored in a context variable (request_id_var) that is automatically maintained across async operations.
3. Logging: All log messages automatically include the request ID via the RequestIdFilter.
4. Outgoing Requests: When using the requests or httpx library, the request ID is automatically injected as the x-request-id header.
5. Response Headers: The request ID is added to the response headers as x-request-id.
6. Celery Tasks: When a Celery task is published, the request ID is automatically included in the task headers and propagated to the worker process. While the task runs, every log line also includes the Celery task_id as celery_task_id.

Log Format

All logs are formatted as JSON with the following structure:

{
  "timestamp": "2024-01-01 12:00:00",
  "level": "INFO",
  "request_id": "550e8400-e29b-41d4-a716-446655440000",
  "path": "/api/users",
  "method": "GET",
  "message": "request received",
  "status_code": 200,
  "error": null,
  "celery_task_id": null
}