opensearch-logger 0.1.0

OpenSearch logging handler

Opensearch Logger for Python

This library provides a standard Python logging handler compatible with Opensearch suite.

The goals of this project are

• to provide a simple and direct logging from Python to Opensearch without fluentd, logstash or other middleware;
• keep it up to date with the growing difference between Opensearch and Elasticsearch projects;
• keep the library easy to use, robust, and simple (entire source code is currently ~300 lines).

The library has been open-sourced from an internal project where it has been successfully used in production since the release of Opensearch 1.0.

Generated log records follow the Elastic Common Schema (ECS) field naming convention. For better performance it is recommended to set up a proper mapping for you logging indices but everything will work even without it. You can find a ready to use compatible JSON mapping in the repository.

Installation

pip install opensearch-logger

Usage

Just add the handler to your logger as follows

import logging
from opensearch_logger import OpensearchHandler

handler = OpensearchHandler(
    hosts=["https://localhost:9200"],
    http_auth=("admin", "admin"),
    use_ssl=True,
    verify_certs=False,
    index_prefix="my-logs",
)

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
logger.addHandler(handler)

To log into Opensearch, simply use the regular logging commands:

logger.info("This message will be indexed in Opensearch")

# Report extra fields
start_time = time.perf_counter()
heavy_database_operation()
elapsed_time = time.perf_counter() - start_time

logger.info(f"Database operation took {elapsed_time:.3f} seconds", extra={"elapsed_time": elapsed_time})

Configuration

The constructor takes the following parameters

Argument	Default	Description
index_prefix	python-logs	Prefix of the Opensearch index name that will be created.
index_rotate	OpensearchHandler.DAILY	The period which is used to generate actual index names and rotate them.
buffer_size	1000	Number of log records which when reached on the internal buffer results in a flush to Opensearch.
flush_frequency	1	Float representing how often the buffer will be flushed (in seconds).
extra_fields	{}	Nested dictionary with all the additional fields that you would like to add to all logs.

All other keyword arguments are passed directly to the underlying Opensearch python client. Full list of connection parameters can be found in opensearch-py docs. Here are few examples:

• hosts: The list of hosts that Opensearch client will connect, multiple hosts are allowed.

  [{"host": "localhost", "port": 9200}, "https://admin:admin@opensearch:9200"]
  

• http_auth: Tuple with user name and password that will be used to authenticate against the Opensearch servers.

  ("admin","admin")
  

• use_ssl: A boolean that defines if the communications should be SSL encrypted.

• verify_certs: A boolean that defines whether the SSL certificates are validated or not.

Configuring using dictConfig or in Django

A complete configuration of logging is also supported. Just specify the opensearch_logger.OpensearchHandler as one of the handlers.

Full guide on tweaking logging.config can be found in the official python documentation for dictConfig.

import logging.config

LOGGING = {
    "version": 1,
    "disable_existing_loggers": True,
    "formatters": {
        "standard": {
            "format": "%(asctime)-15s | %(process)d | %(levelname)s | %(name)s | %(message)s",
            "datefmt": "%Y-%m-%d %H:%M:%S %z",
        },
    },
    "handlers": {
        "console": {
            "level": "INFO",
            "formatter": "standard",
            "class": "logging.StreamHandler",
            "stream": "ext://sys.stderr",
        },
        "file": {
            "level": "DEBUG",
            "class": "logging.handlers.RotatingFileHandler",
            "filename": "./debug.log",
            "maxBytes": 102400,
            "backupCount": 5,
        },
        "opensearch": {
            "level": "INFO",
            "class": "opensearch_logger.OpensearchHandler",
            "hosts": [{"host": "localhost", "port": 9200}],
            "index_prefix": "my-logs",
            "extra_fields": {"App": "test", "Environment": "dev"},
            "use_ssl": True,
            "verify_certs": False,
        },
    },
    "loggers": {
        "root": {
            "handlers": ["console", "file", "opensearch"],
            "level": "INFO",
            "propogate": False,
        },
        "django": {
            "handlers": ["file","opensearch"],
            "level": "DEBUG",
            "propagate": True,
        },
    },
}

logging.config.dictConfig(LOGGING)

In the example above following things are configured:

• 1 formatter named "standard" that will be used to output to the terminal.
• 3 different log handlers:
  • One named console for logging to terminal, through the stderr stream
  • One called file for logging into a rotated log file
  • And finally one for Opensearch
• 2 loggers are configured:
  • root, from which all other loggers are derived, will log all messages with level INFO or higher using all three handlers.
  • django handler log all messages with level DEBUG or higher using file and opensearch handlers.

Dependencies

This library uses the following packages

• opensearch-py

Building from source & Developing

This package uses pyenv (optional) and Poetry for development purposes. It also uses Docker to run Opensearch container for integration testing during development.

1. Clone the repo

2. Instruct poetry to use a proper Python version for virtual environment creation.

   poetry env use 3.8.12
   

3. Create virtual environment and install dependencies

   poetry install
   

4. Test that the library works

   WARNING: You need Docker installed and available on the system to run the tests. Part of the tests verifies that logs actually get into Opensearch that runs in a docker container.

   poetry run pytest
   

5. Build a package

   poetry build
   

Contributions

Contributions are welcome! 👏 🎉

Please create a GitHub issue and a Pull Request that references that issue as well as your proposed changes. Your Pull Request will be automatically tested using GitHub actions.

After your pull request will be accepted, it will be merged and the version of the library will be bumped and released to PyPI.