A simple logger for multiple Python processes.
Project description
A simple, multiprocess-safe logger for Python
Why
Python’s built-in loggers are pretty handy - they’re easily customized and come with useful functionality out of the box, including things like file rotation. These file handlers are thread-safe, but not process-safe, so, if you’re running a webserver in a pre-forking environment, for example, you run the risk of your workers trampling over each other when writing to a common log file. File locking is a possible workaround, but that’s yucky.
To avoid this, it is recommended that one uses a socket-based logger (a code sample is helpfully provided in the Logging Cookbook). However, it is just a code snippet. Multilog is a dependency-free implementation of the sample socket logger with some niceties, like fileConfig support, and parameterization.
How
Once installed, the Multilog daemon can be invoked via:
mutlilog
Usage:
usage: multilog [-h] [-s SERVER] [-p PORT] [-c CONFIG_PATH]
A simple logger for multiple Python processes.
optional arguments:
-h, --help show this help message and exit
-s SERVER, --server SERVER
The server hostname (default: localhost)
-p PORT, --port PORT The port to listen on. (default: 9020)
-c CONFIG_PATH, --config CONFIG_PATH
The log configuration to load. (default: logging.ini)
By default, it will look for a logging.ini file in the current directory. If one isn’t found, Multilog will yell at you. A sample configuration file for the server:
[loggers] keys=root [handlers] keys=multilogServerHandler [formatters] keys=simpleFormatter [logger_root] level=NOTSET handlers=multilogServerHandler [handler_multilogServerHandler] class=handlers.TimedRotatingFileHandler level=DEBUG formatter=simpleFormatter args=('/var/log/appName/appName.log', 'midnight') [formatter_simpleFormatter] class=logging.Formatter format=%(asctime)s %(levelname)7s: PID: %(process)5s | %(message)s [in %(pathname)s:%(lineno)d]
and for your application:
[loggers] keys=root [handlers] keys=multilogClientHandler [formatters] keys=simpleFormatter [logger_root] level=NOTSET handlers=multilogClientHandler [handler_multilogClientHandler] class=handlers.SocketHandler level=DEBUG formatter=simpleFormatter args=('localhost', handlers.DEFAULT_TCP_LOGGING_PORT) [formatter_simpleFormatter] class=logging.Formatter format=%(asctime)s %(levelname)7s: PID: %(process)5s | %(message)s [in %(pathname)s:%(lineno)d]
The important field is the args block in the handler_multilogClientHandler section - those parameters should correspond to the server and ports on which the multilog daemon is listening. By default, the daemon uses localhost and logging.handlers.DEFAULT_TCP_LOGGING_PORT.
For Power Users
If you want to have Multilog share your application’s config, you can do the following:
[loggers] keys=root,appName [handlers] keys=multilogClientHandler,multilogServerHandler [formatters] keys=simpleFormatter [logger_root] level=NOTSET handlers=%(root_handler)s [logger_appName] level=INFO handlers= propagate=1 qualname=appName [handler_multilogClientHandler] class=handlers.SocketHandler level=DEBUG formatter=simpleFormatter args=('localhost', handlers.DEFAULT_TCP_LOGGING_PORT) [handler_multilogServerHandler] class=handlers.TimedRotatingFileHandler level=DEBUG formatter=simpleFormatter args=('/var/log/appName/appName.log', 'midnight') [formatter_simpleFormatter] class=logging.Formatter format=%(asctime)s %(levelname)7s: PID: %(process)5s | %(message)s [in %(pathname)s:%(lineno)d]
Then, in your application, pass the root handler name into the logging config:
import logging logging.config.fileConfig(config_path, defaults={"root_handler": "multilogClientHandler"})
Multilog will always load the multilogServerHandler handler. If you don’t want to run Multilog (if you’re running a single-threaded local dev server, for example), simply change your root_handler value to multilogServerHandler to write to the handler.
Support
Multilog is compatible with Python 2.6, 2.7, and 3.3+.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.