Skip to main content

A Python logging handler that sends your logs to Splunk

Project description

Splunk Handler

Build Code Climate PyPI

Splunk Handler is a Python Logger for sending logged events to an installation of Splunk Enterprise.

This logger requires the destination Splunk Enterprise server to have enabled and configured the Splunk HTTP Event Collector.

A Note on Using with AWS Lambda

AWS Lambda has a custom implementation of Python Threading, and does not signal when the main thread exits. Because of this, it is possible to have Lambda halt execution while logs are still being processed. To ensure that execution does not terminate prematurely, Lambda users will be required to invoke splunk_handler.force_flush directly as the very last call in the Lambda handler, which will block the main thread from exiting until all logs have processed.

from splunk_handler import force_flush

def lambda_handler(event, context):
    do_work()
    force_flush()  # Flush logs in a blocking manner

Installation

Pip:

pip install splunk_handler

Manual:

python setup.py install

Usage

from splunk_handler import SplunkHandler

Then use it like any other regular Python logging handler.

Example:

    import logging
    from splunk_handler import SplunkHandler
    splunk = SplunkHandler(
        host='splunk.example.com',
        port='8088',
        token='851A5E58-4EF1-7291-F947-F614A76ACB21',
        index='main'
        #allow_overrides=True # whether to look for _<param in log data (ex: _index)
        #debug=True # whether to print module activity to stdout, defaults to False
        #flush_interval=15.0, # send batch of logs every n sec, defaults to 15.0, set '0' to block thread & send immediately
        #force_keep_ahead=True # sleep instead of dropping logs when queue fills
        #hostname='hostname', # manually set a hostname parameter, defaults to socket.gethostname()
        #protocol='http', # set the protocol which will be used to connect to the splunk host
        #proxies={
        #           'http': 'http://10.10.1.10:3128',
        #           'https': 'http://10.10.1.10:1080',
        #         }, set the proxies for the session request to splunk host
        #
        #queue_size=5000, # a throttle to prevent resource overconsumption, defaults to 5000, set to 0 for no max
        #record_format=True, whether the log format will be json
        #retry_backoff=1, the requests lib backoff factor, default options will retry for 1 min, defaults to 2.0
        #retry_count=5, number of retry attempts on a failed/erroring connection, defaults to 5
        #source='source', # manually set a source, defaults to the log record.pathname
        #sourcetype='sourcetype', # manually set a sourcetype, defaults to 'text'
        #verify=True, # turn SSL verification on or off, defaults to True
        #timeout=60, # timeout for waiting on a 200 OK from Splunk server, defaults to 60s
    )

    logging.getLogger('').addHandler(splunk)

    logging.warning('hello!')

I would recommend using a JSON formatter with this to receive your logs in JSON format. Here is an open source one: https://github.com/madzak/python-json-logger

Logging Config

Sometimes it's a good idea to create a logging configuration using a Python dict and the logging.config.dictConfig function. This method is used by default in Django.

Here is an example dictionary config and how it might be used in a settings file:

import os

# Splunk settings
SPLUNK_HOST = os.getenv('SPLUNK_HOST', 'splunk.example.com')
SPLUNK_PORT = int(os.getenv('SPLUNK_PORT', '8088'))
SPLUNK_TOKEN = os.getenv('SPLUNK_TOKEN', '851A5E58-4EF1-7291-F947-F614A76ACB21')
SPLUNK_INDEX = os.getenv('SPLUNK_INDEX', 'main')

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'json': {
            '()': 'pythonjsonlogger.jsonlogger.JsonFormatter',
            'format': '%(asctime)s %(created)f %(exc_info)s %(filename)s %(funcName)s %(levelname)s %(levelno)s %(lineno)d %(module)s %(message)s %(pathname)s %(process)s %(processName)s %(relativeCreated)d %(thread)s %(threadName)s'
        }
    },
    'handlers': {
        'splunk': {
            'level': 'DEBUG',
            'class': 'splunk_handler.SplunkHandler',
            'formatter': 'json',
            'host': SPLUNK_HOST,
            'port': SPLUNK_PORT,
            'token': SPLUNK_TOKEN,
            'index': SPLUNK_INDEX,
            'sourcetype': 'json',
        },
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
        }
    },
    'loggers': {
        '': {
            'handlers': ['console', 'splunk'],
            'level': 'DEBUG'
        }
    }
}

Then, do logging.config.dictConfig(LOGGING) to configure your logging.

Note: I included a configuration for the JSON formatter mentioned above.

Here is an example file config, and how it might be used in a config file:

[loggers]
keys=root

[handlers]
keys=consoleHandler,splunkHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=%(loglevel)s
handlers=consoleHandler,splunkHandler

[handler_consoleHandler]
class=StreamHandler
level=%(loglevel)s
formatter=simpleFormatter
args=(sys.stdout,)

[handler_splunkHandler]
class=splunk_handler.SplunkHandler
level=%(loglevel)s
formatter=simpleFormatter
args=('my-splunk-host.me.com', '', os.environ.get('SPLUNK_TOKEN_DEV', 'changeme'), 'my_index')
kwargs={'url':'https://my-splunk-host.me.com/services/collector/event', 'verify': False}

[formatter_simpleFormatter]
format=[%(asctime)s] %(levelname)s - %(module)s: %(message)s
datefmt=%m/%d/%Y %I:%M:%S %p %Z

Retry Logic

This library uses the built-in retry logic from urllib3 (a retry counter and a backoff factor). Should the defaults not be desireable, you can find more information about how to best configure these settings in the urllib3 documentation.

Contributing

Feel free to contribute an issue or pull request:

  1. Check for existing issues and PRs
  2. Fork the repo, and clone it locally
  3. Create a new branch for your contribution
  4. Push to your fork and submit a pull request

License

This project is licensed under the terms of the MIT license.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

splunk_handler-3.0.0.tar.gz (11.0 kB view details)

Uploaded Source

Built Distribution

splunk_handler-3.0.0-py2.py3-none-any.whl (9.0 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file splunk_handler-3.0.0.tar.gz.

File metadata

  • Download URL: splunk_handler-3.0.0.tar.gz
  • Upload date:
  • Size: 11.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.4 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.1 CPython/3.9.6

File hashes

Hashes for splunk_handler-3.0.0.tar.gz
Algorithm Hash digest
SHA256 a835b0a7161148aceb977301c6dba21bb96e7635279c7e90e0216c78a4bcbb24
MD5 54e0761fdf5823104a3796cc500620b7
BLAKE2b-256 32fc23accd936e52a6d2eb75db17bf0002177f01df9e460fed3f77116fe1a844

See more details on using hashes here.

File details

Details for the file splunk_handler-3.0.0-py2.py3-none-any.whl.

File metadata

  • Download URL: splunk_handler-3.0.0-py2.py3-none-any.whl
  • Upload date:
  • Size: 9.0 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.4 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.1 CPython/3.9.6

File hashes

Hashes for splunk_handler-3.0.0-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 76e85ee70c6ca00c6d8be1f64715653fc00af3c97853b70d91e156704dbc8486
MD5 f74ed108a3c41624e700d00056494bad
BLAKE2b-256 2ab2186b49e839839fc51d4177dab61e0c08cb37dd477b5e41d34b3c800f9104

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page