A system log management tool with automatically generated log templates.
Project description
Amulog is a tool to support system log management. The main function is to classify log messages with automatically generated log templates (formats and variable locations), and to store the data in a database. This system works on python3.
Source: https://github.com/cpflat/amulog
Bug Reports: https://github.com/cpflat/amulog/issues
Author: Satoru Kobayashi
License: BSD-3-Clause
Main features
Support multiple databases: sqlite and mysql
Smart log segmentation with log2seq
Multiple template generation algorithms like: Drain, SHISO, LenMa, FT-tree, Dlog, etc.
Import/Export log templates
Edit log templates manually
Search API with datetime, hostname and log template IDs
Tutorial
Install
$ pip install amulog
Generate config
For the first step, save following config as test.conf
on an empty directory.
[general] src_path = logfile.txt src_recur = false logging = auto.log [database] database = sqlite3 sqlite3_filename = log.db [log_template] lt_methods = drain indata_filename = ltgen.dump
Then modify general.src_path
option to a logfile you want to load.
(If you want to use multiple files, change general.src_recur
into true and specify directory name to general.src_path
.)
#### Generate database
Try following command to generate database:
$ python -m amulog db-make -c test.conf
Check database
$ python -m amulog show-db-info -c test.conf
shows status of the generated database.
$ python -m amulog show-lt -c test.conf
shows all generated log templates in the given logfile.
$ python -m amulog show-log -c test.conf ltid=2
shows all log messages corresponding to log template ID 2.
Resume generating database
Try following command to resume generating database:
$ python -m amulog db-add -c test.conf logfile2.txt
Export and Import templates
Following command exports all log templates in the database:
$ python3 -m amulog show-db-import -c test.conf > exported_tpl.txt
You can modify the exported templates manually.
Note that some special letters (\,@,*
) are escaped in the exported templates.
To import the templates, save following config as test2.conf
.
[general] src_path = logfile.txt src_recur = false logging = new_auto.log [database] database = sqlite3 sqlite3_filename = new_log.db [log_template] lt_methods = import indata_filename = new_ltgen.dump [log_template_import] def_path = exported_tpl.txt
Then, try generating database again:
python -m amulog db-make -c test2.conf
Further usage
see help with following command:
python -m amulog -h
Reference
This tool is demonstrated at CNSM2020. If you use this code, please consider citing:
@inproceedings{Kobayashi_CNSM2020, author = {Kobayashi, Satoru and Yuya, Yamashiro and Otomo, Kazuki and Fukuda, Kensuke and Esaki, Hiroshi}, booktitle = {Proceedings of the 16th International Conference on Network and Service Management (CNSM'20)}, title = {amulog: A General Log Analysis Framework for Diverse Template Generation Methods}, year = {2020} }
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.