Skip to main content

A system log management tool with automatically generated log templates.

Project description

PyPI release Python support BSD 3-Clause License Travis CI

Amulog is a tool to support system log management. The main function is to classify log messages with automatically generated log templates (formats and variable locations), and to store the data in a database. This system works on python3.

Main features

  • Support multiple databases: sqlite and mysql
  • Smart log segmentation with log2seq
  • Multiple template generation algorithms such as: Drain, SHISO, LenMa, FT-tree, Dlog, etc.
  • Support Online (incremental) and Offline (hindsight) use
  • Suspend and resume the template generation process
  • Import and Export log templates if you need
  • Edit log templates manually if you need
  • Search API with datetime, hostname and log template IDs

Tutorial

Install

$ pip install amulog

Generate config

For the first step, save following config as test.conf on an empty directory.

[general]
src_path = logfile.txt
src_recur = false
logging = auto.log

[database]
database = sqlite3
sqlite3_filename = log.db

[log_template]
lt_methods = drain
indata_filename = ltgen.dump

Then modify general.src_path option to a logfile you want to load. (If you want to use multiple files, change general.src_recur into true and specify directory name to general.src_path.)

Generate database

Try following command to generate database:

$ python -m amulog db-make -c test.conf

Check database

$ python -m amulog show-db-info -c test.conf

shows status of the generated database.

$ python -m amulog show-lt -c test.conf

shows all generated log templates in the given logfile.

$ python -m amulog show-log -c test.conf ltid=2

shows all log messages corresponding to log template ID 2.

Resume generating database

Try following command to resume generating database:

$ python -m amulog db-add -c test.conf logfile2.txt

Export and Import templates

Following command exports all log templates in the database:

$ python3 -m amulog show-db-import -c test.conf > exported_tpl.txt

You can modify the exported templates manually. Note that some special letters (\\, @, *) are escaped in the exported templates.

To import the templates, save following config as test2.conf.

[general]
src_path = logfile.txt
src_recur = false
logging = new_auto.log

[database]
database = sqlite3
sqlite3_filename = new_log.db

[log_template]
lt_methods = import
indata_filename = new_ltgen.dump

[log_template_import]
def_path = exported_tpl.txt

Then, try generating database again:

python -m amulog db-make -c test2.conf

Further usage

see help with following command:

python -m amulog -h

Reference

This tool is demonstrated at CNSM2020. If you use this code, please consider citing:

@inproceedings{Kobayashi_CNSM2020,
  author = {Kobayashi, Satoru and Yamashiro, Yuya and Otomo, Kazuki and Fukuda, Kensuke},
  booktitle = {Proceedings of the 16th International Conference on Network and Service Management (CNSM'20)},
  title = {amulog: A General Log Analysis Framework for Diverse Template Generation Methods},
  pages={1-5},
  year = {2020}
}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for amulog, version 0.1.12
Filename, size File type Python version Upload date Hashes
Filename, size amulog-0.1.12.tar.gz (91.1 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page