Library for wrapping cluster logging for Altamira projects.
Project description
Cluster Logging library for python 3.6
This is a library for python 3.6 that will help fulfil a requirement for cluster services and logging to elastic search.
Requirements
- Python 3.6+
- pipenv
- Elasticsearch
- Fluentd
- Kibana
Pipenv & Pipfile
Pipenv combines package management and virtualenv into one tool.
pipenv: https://docs.pipenv.org/basics
To install:
$ pip install pipenv
Pipfile replaces requirements.txt and it will specify both dependencies and dev dependencies in one file.
Development Installation
# install dependencies
pipenv install --dev
# activate virtual environment
pipenv shell
# alternativly you can start your script in a virtual environment context
pipenv run python main.py message that you want to send
Standalone Usage (Development)
It may be required to run the following command to get elastic search to run.
sudo sysctl -w vm.max_map_count=262144
Local
$ pipenv run python main.py -h
usage: main.py [-h] [-n HOST] [-p PORT] [-e [ENV [ENV ...]]] [-o PROJECT]
[-a APP] [-c COUNT]
Console Application to test Cluster Logging using fluentd.
positional arguments:
message Message to be logged.
optional arguments:
-h, --help show this help message and exit
-n HOST, --host HOST Hostname to listen on. (default=localhost)
-p PORT, --port PORT Port number to bind to. (default=24224)
-e [ENV [ENV ...]], --env [ENV [ENV ...]]
Environment Keys to add to message.
-o PROJECT, --project PROJECT
Tag for the project
-a APP, --app APP Tag for the application
-c COUNT, --count COUNT
The number of times to send message.
Docker
$ docker build -t TAG_FOR_THE_BUILD:VERSION .
# all env values can be set with -e
$ docker run -e host=fluentd \
-e port=24224 -e msg="message you want to send" TAG_FOR_THE_BUILD:VERSION
# If you want to run the whole environment you can use docker-compose up
$ docker-compose up
Api
Follows the python logger implementation with logging levels. python.org
Level | Numeric value |
---|---|
CRITICAL | 50 |
ERROR | 40 |
WARNING | 30 |
INFO | 20 |
DEBUG | 10 |
NOTSET | 0 |
Logging level can be set for logging using:
import logging
logging.basicConfig(level=logging.WARNING)
Setting the logging level will filter levels below the set logging level. For logging levels error and above the stack trace will be added to the log message.
Configuration
These environment variables can be set either system-wide or added to the .env
file:
There are 3 Environment variables that will be extracted by default:
HOST
- host that the container is running on.MARATHON_APP_ID
- The ID that is assigned by DCOS.MARATHON_APP_DOCKER_IMAGE
- The image that the container was built from.
There are three ways to get application properties into log messages.
- pass a dictionary of key value pairs
- pass a list of keys that are on the container environment
- or pass a JSON file with key value pairs.
NOTE:
- The dictionary will overwrite values in environment and JSON file.
- The Environment will overwrite values in the JSON file.
Usage
Activate your venv
# Linux / MacOS
source {NAME_OF_VENV}/bin/activate
REM Windows
{NAME_OF_VENV}\Scripts\activate
Install cluster logger into your venv. Cluster Logger has been installed into pypi.
$ pip install cluster_logger
To Use
import cluster_logger
import logging
# Then init the ClusterLogger class
env_keys = ['MARATHON_APP_LABELS', 'MARATHON_APP_RESOURCE_CPUS']
props = {'HOST', '192.168.0.12'}
logging.basicConfig(level=logging.DEBUG) # set logging level
cluster_logger.initLogger('Rasters', # Project name
'intent_client', # Application name
'fluentd', # Fluentd Host name (optional) will default to fluentd if nothing is passed in and HOST isn't set on the environment
24224, # Fluentd Port Number(optional)
'/path/to/settings.json', # JSON settings file (optional)
env_keys, # Environment Keys (optional)
props) # Specific Properties (optional)
# in Module where you will log
import cluster_logger
logger = cluster_logger.getLogger(__name__)
logger.exception('Custom Message to send typically an error')
Contributing
# bump the version that is stored in setup.py and cluster_logger/__init__.py
$ pipenv run bumpversion minor # possible: major / minor / patch
$ git push
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cluster_logger-0.4.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | af52898e197e795b58cb6e53cfec4a9665501431139b39e44e55afd444288294 |
|
MD5 | 00e271356cd27ab9b0faa8c60e4d7575 |
|
BLAKE2b-256 | bfcc9afed9baed5eb57a56b5c6740c4ea08f12115537f97c4f930bb9ea9be551 |