phanos 0.3.2

Python client to gather data for Prometheus logging in server with multiple instances and workers.

PHANOS

Python client to gather data for Prometheus logging in server with multiple instances and workers. Phanos works with synchronous programs and with asynchronous programs implemented by asyncio. Behavior in multithread programs is unspecified.

Profiling

Default metrics

Phanos contains two default metrics. Time profiler measuring execution time of decorated methods is ON by default. Response size profiler measuring size of methods return value of root method is OFF by default. Both can be deleted by phanos.profiler.delete_metric(phanos.publisher.TIME_PROFILER) and phanos.profiler.delete_metric(phanos.publisher.RESPONSE_SIZE).

Configuration

There are two options of how to configure profiler. Profiler must be configured by one of these options.

Both options uses these attributes:

• job job label for prometheus; usually name of app

• logger (optional) name of logger; if not specified, Phanos creates its own logger

• time_profile (optional) by default profiler tracks execution time of profiled function/object

• response_size_profile (optional) by default profiler will not track size of return value

• handle_records (optional) by default profiler measures values and handles them; if False then no profiling is made or handled

• error_raised_label (optional) by default profiler will not add label error_raised to each record; if set, every record will have information if given profiled function/method raised error; if error raised, profiling will be logged even if no LoggerHandler exists

• handlers (optional) serialized named handlers to publish profiled records; if no handlers specified then no measurements are made; for handlers description refer to Handlers.

  • class class handler to initialized
  • handler_name handler name required argument of publishers
  • **other arguments - specific arguments required to construct instance of class f.e.: output.

With addition of async handlers, async_config and async_dict_config methods were added to Profiler class. Be wary that config and dict_config methods ARE NOT compatible with async handlers, but async_config and async_dict_config ARE compatible with sync handlers, but cannot be used in purely synchronise environment.

Dict Configuration

It is possible to configure profile with configration dictionary with method Profiler.dict_config(settings) or Profiler.async_dict_config(settings) for usage with async handlers. (similar to logging dictConfig).

Example of configuration dict:

settings = {
    "job": "my_app", 
    "logger": "my_app_debug_logger", 
    "time_profile": True, 
    "response_size_profile": False,
    "handle_records": True, 
    "error_raised_label": False,
    "handlers": {
        "stdout_handler_ref": {
                "class": "phanos.handlers.StreamHandler", 
                "handler_name": "stdout_handler", 
                "output": "ext://sys.stdout"
            }
        }
}

Configuration in code

When configuring in code use Profiler.config or Profiler.async_config method to configure profiling. For handler addition create handler instance first and add it to profiler with Profiler.add_handler method. (Cannot add_handler before profiler configuration)

Example of configuration:

import logging
import phanos
    
# configuration of profiler and handler addition
phanos.profiler.config(
  logger=None, 
  time_profile=True, 
  response_size_profile=False, 
  handle_records=True, 
  error_raised_label=True,
)
log_handler = phanos.publisher.LoggerHandler(
  'handler_name', 
  logger=None, 
  level=logging.INFO
)
phanos.profiler.add_handler(log_handler)    
        

Usage

• configure profiler as shown in Configuration

• decorate methods which you want to profile @phanos.profile. (shortcut for @phanos.profiler.profile). Allways put decorator closest to method definition as possible, because the decorator collides with some other decorators. The decorator must allways be under @classmethod, @staticmethod, flask_restx decorators and possibly others.

import phanos
   
class SomeClass:
    """ Example of @profile decorator placement"""
    @staticmethod
    @phanos.profile
    def some_method():
        pass
    
    # is equivalent to
    @staticmethod
    @phanos.profiler.profile
    def some_method():
        pass

Handlers

Each handler have handler_name attribute. This string can be used to delete handlers later with phanos.profiler.delete_handler(handler_name).

Records can be handled by these handlers:

• StreamHandler(handler_name, output) - write records to given output (default is sys.stdout)
• LoggerHandler(handler_name, logger, level) - logs string representation of records with given logger and with given level; default level is logging.DEBUG; if no logger is passed, Phanos creates its own logger
• NamedLoggerHandler(handler_name, logger_name, level) - same as LoggerHandler, but logger instance is found by logging.getLogger(logger_name) method.
• ImpProfHandler(handler_name, **rabbit_connection_params, logger) - sending records to RabbitMQ queue - blocking.
• AsyncImpProfHandler(handler_name, **rabbit_connection_params, logger) - sending records to RabbitMQ queue - async.

Phanos metrics:

Basic Prometheus metrics:

• Histogram
• Summary
• Counter
• Info
• Gauge
• Enum

These classes represent basic Prometheus metrics types. For more information about Prometheus metric types, allowed operations, etc. refer to Prometheus documentation.

Custom metrics

• TimeProfiler: metric for measuring time-consuming actions in mS; basically Histogram metric of Prometheus.
• ResponseSize: metric for measuring size of return value of method in bytes, designed to measure response size of endpoints; basically Histogram metric of Prometheus.

Creating new custom metric

• new metric class needs to inherit from one of basic Prometheus metrics.
• __init__() method needs to call super().__init__()
• implement method for each operation wanted; this method must call one of inherited metrics operations if you want operation to be stored f.e. Gauge.dec;
• MetricWrapper.cleanup() is called after all measured metrics are handled; if custom cleanup is needed, implement method cleanup() calling super().cleanup() inside

Add metrics automatic measurements

phanos.profiler contains these four attributes:

• before_func: Callable[[Callable[..., Any], Tuple[Any, ...], Dict[str, Any]], None]: executes before each profiled method/function
• before_root_func: Callable[[Callable[..., Any], Tuple[Any, ...], Dict[str, Any]], None]: executes before each profiled root method/function (first method in profiling tree)
• after_func: Callable[[Any, Tuple[Any, ...], Dict[str, Any]], None]: executes after each profiled method/function
• after_root_func: Callable[[Any, Tuple[Any, ...], Dict[str, Any]], None]: executes after each profiled root method/function (first method in profiling tree)

Implement these methods with all needed measurement.

Complete example

import typing
import phanos


# custom metric example
class CustomMetric(phanos.metrics.Counter):
  def __init__(self, name, job, units, labels):
    super().__init__(name, job, units, labels)
    # custom initialization
    self.count = 0

  def custom_op(self, value: int = 0, label_values: typing.Optional[typing.Dict[str, str]] = None):
    self.count += value
    self.inc(self.count, label_values)

  def helper_method(self):
    pass

  def cleanup(self) -> None:
    super().cleanup()
    self.count = 0


my_metric = CustomMetric(name="name", job="MyJob", units="units", labels={"label_name"})


def before_function(func, args, kwargs):
  # this operation will be recorded
  my_metric.custom_op(2, {"label_name": "label_value"})
  # this won't be recorded
  my_metric.helper_method()


phanos.profiler.before_func = before_function

What must/can be done:

• custom metric
  • MetricWrapper.__init__ needs name, job, units arguments passed; labels and logger are optional
• custom measurements
  • before_* functions must have func argument, where function which is executed is passed. after_* function needs to have result argument where function result is passed
  • all four functions can access args and kwargs of decorated methods. These arguments are passed in packed form.