amulog 0.3.10

A system log management tool with automatically generated log templates.

Amulog is a tool to support system log management. The main function is to classify log messages with automatically generated log templates (formats and variable locations), and to store the data in a database. This system works on python3.

• Source: https://github.com/amulog/amulog

• Bug Reports: https://github.com/amulog/amulog/issues

• Author: Satoru Kobayashi

• License: BSD-3-Clause

Main features

• Support multiple databases: sqlite and mysql

• Smart log segmentation with log2seq

• Multiple template generation algorithms such as: Drain, SHISO, LenMa, FT-tree, Dlog, etc.

• Support Online (incremental) and Offline (hindsight) use

• Suspend and resume the template generation process

• Import and Export log templates if you need

• Edit log templates manually if you need

• Search API with datetime, hostname and log template IDs

Tutorial

Install

$ pip install amulog

Generate config

For the first step, save following config as test.conf on an empty directory.

[general]
src_path = logfile.txt
src_recur = false
logging = auto.log

[database]
database = sqlite3
sqlite3_filename = log.db

[log_template]
lt_methods = drain
indata_filename = ltgen.dump

Then modify general.src_path option to a logfile you want to load. (If you want to use multiple files, change general.src_recur into true and specify directory name to general.src_path.)

Generate database

Try following command to generate database:

$ python -m amulog db-make -c test.conf

Check database

$ python -m amulog show-db-info -c test.conf

shows status of the generated database.

$ python -m amulog show-lt -c test.conf

shows all generated log templates in the given logfile.

$ python -m amulog show-log -c test.conf ltid=2

shows all log messages corresponding to log template ID 2.

Resume generating database

Try following command to resume generating database:

$ python -m amulog db-add -c test.conf logfile2.txt

Export and Import templates

Following command exports all log templates in the database:

$ python3 -m amulog show-db-import -c test.conf > exported_tpl.txt

You can modify the exported templates manually. Note that some special letters (\\, @, *) are escaped in the exported templates.

To import the templates, save following config as test2.conf.

[general]
src_path = logfile.txt
src_recur = false
logging = new_auto.log

[database]
database = sqlite3
sqlite3_filename = new_log.db

[log_template]
lt_methods = import
indata_filename = new_ltgen.dump

[log_template_import]
def_path = exported_tpl.txt

Then, try generating database again:

python -m amulog db-make -c test2.conf

Further usage

see help with following command:

python -m amulog -h

Reference

This tool is demonstrated at International Journal of Network Management and CNSM2020.

If you use this code, please consider citing:

@article{Kobayashi_IJNM2022,
  author = {Kobayashi, Satoru and Yamashiro, Yuya and Otomo, Kazuki and Fukuda, Kensuke},
  title = {amulog: A general log analysis framework for comparison and combination of diverse template generation methods*},
  journal = {International Journal of Network Management},
  volume = {32},
  number = {4},
  pages = {e2195},
  doi = {https://doi.org/10.1002/nem.2195},
  year = {2022}
}

@inproceedings{Kobayashi_CNSM2020,
  author = {Kobayashi, Satoru and Yamashiro, Yuya and Otomo, Kazuki and Fukuda, Kensuke},
  booktitle = {Proceedings of the 16th International Conference on Network and Service Management (CNSM'20)},
  title = {amulog: A General Log Analysis Framework for Diverse Template Generation Methods},
  pages={1-5},
  year = {2020}
}