Skip to main content

A simple logger for multiple Python processes.

Project description

https://coveralls.io/repos/github/humangeo/multilog/badge.svg?branch=master

A simple, multiprocess-safe logger for Python

Why

Python’s built-in loggers are pretty handy - they’re easily customized and come with useful functionality out of the box, including things like file rotation. These file handlers are thread-safe, but not process-safe, so, if you’re running a webserver in a pre-forking environment, for example, you run the risk of your workers trampling over each other when writing to a common log file. File locking is a possible workaround, but that’s yucky.

To avoid this, it is recommended that one uses a socket-based logger (a code sample is helpfully provided in the Logging Cookbook). However, it is just a code snippet. Multilog is a dependency-free implementation of the sample socket logger with some niceties, like fileConfig support, and parameterization.

How

Once installed, the Multilog daemon can be invoked via:

mutlilog

Usage:

usage: multilog [-h] [-s SERVER] [-p PORT] [-c CONFIG_PATH]

A simple logger for multiple Python processes.

optional arguments:
  -h, --help                    show this help message and exit
  -s SERVER, --server SERVER
                                The server hostname (default: localhost)
  -p PORT, --port PORT          The port to listen on. (default: 9020)
  -c CONFIG_PATH, --config CONFIG_PATH
                                The log configuration to load. (default: logging.ini)

By default, it will look for a logging.ini file in the current directory. If one isn’t found, Multilog will yell at you. A sample configuration file for the server:

[loggers]
keys=root

[handlers]
keys=multilogServerHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=NOTSET
handlers=multilogServerHandler

[handler_multilogServerHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=simpleFormatter
args=('/var/log/appName/appName.log', 'midnight')

[formatter_simpleFormatter]
class=logging.Formatter
format=%(asctime)s %(levelname)7s: PID: %(process)5s | %(message)s [in %(pathname)s:%(lineno)d]

and for your application:

[loggers]
keys=root

[handlers]
keys=multilogClientHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=NOTSET
handlers=multilogClientHandler

[handler_multilogClientHandler]
class=handlers.SocketHandler
level=DEBUG
formatter=simpleFormatter
args=('localhost', handlers.DEFAULT_TCP_LOGGING_PORT)

[formatter_simpleFormatter]
class=logging.Formatter
format=%(asctime)s %(levelname)7s: PID: %(process)5s | %(message)s [in %(pathname)s:%(lineno)d]

The important field is the args block in the handler_multilogClientHandler section - those parameters should correspond to the server and ports on which the multilog daemon is listening. By default, the daemon uses localhost and logging.handlers.DEFAULT_TCP_LOGGING_PORT.

For Power Users

If you want to have Multilog share your application’s config, you can do the following:

[loggers]
keys=root,appName

[handlers]
keys=multilogClientHandler,multilogServerHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=NOTSET
handlers=%(root_handler)s

[logger_appName]
level=INFO
handlers=
propagate=1
qualname=appName

[handler_multilogClientHandler]
class=handlers.SocketHandler
level=DEBUG
formatter=simpleFormatter
args=('localhost', handlers.DEFAULT_TCP_LOGGING_PORT)

[handler_multilogServerHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=simpleFormatter
args=('/var/log/appName/appName.log', 'midnight')

[formatter_simpleFormatter]
class=logging.Formatter
format=%(asctime)s %(levelname)7s: PID: %(process)5s | %(message)s [in %(pathname)s:%(lineno)d]

Then, in your application, pass the root handler name into the logging config:

import logging
logging.config.fileConfig(config_path, defaults={"root_handler": "multilogClientHandler"})

Multilog will always load the multilogServerHandler handler. If you don’t want to run Multilog (if you’re running a single-threaded local dev server, for example), simply change your root_handler value to multilogServerHandler to write to the handler.

Support

Multilog is compatible with Python 2.6, 2.7, and 3.3+.

Project details


Release history Release notifications

This version
History Node

1.1.0

History Node

1.0.0

History Node

0.9.0

History Node

0.9.0dev

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
multilog-1.1.0.tar.gz (5.3 kB) Copy SHA256 hash SHA256 Source None Feb 8, 2016

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page