Skip to main content

Auto parser for system raw log without human guidance

Project description

Spell

Use python implement the paper Spell: Streaming Parsing of System Event Logs from Min Du, Feifei Li @University of Utah.

This implement is refactored and enhancement version of logpai's logparser.

Install

pip install spellpy

Getting Started

python example.py

After executing the line above, the result folder will be created and you will see two files: structured.csv and templates.csv.

 

  • *_main_structured.csv
... Level Component Content EventId EventTemplate ParameterList
... INFO dfs.DataNode$DataXceiver Receiving block blk_-1608999687919862906 src: /10.250.19.102:54106 dest: /10.250.19.102:50010 f57d69cf Receiving block blk_-1608999687919862906 src <*> <*> dest <*> 50010 ['/10.250.19.102:54106', '/10.250.19.102']
... INFO dfs.DataNode$PacketResponder PacketResponder 1 for block blk_-1608999687919862906 terminating 7b619377 PacketResponder <*> for block blk_-1608999687919862906 terminating ['1']
... INFO dfs.DataNode$DataXceiver Receiving block blk_-1608999687919862906 src: /10.250.10.6:40524 dest: /10.250.10.6:50010 f57d69cf Receiving block blk_-1608999687919862906 src <*> <*> dest <*> 50010 ['/10.250.10.6:40524', '/10.250.10.6']

 

  • *_main_templates.csv
EventId EventTemplate Occurrences
6af214fd Receiving block <*> src <*> <*> dest <*> 50010 5
26ae4ce0 BLOCK* NameSystem.allocateBlock <*> 2
dc2c74b7 PacketResponder <*> for block <*> terminating 4

 

As you see, there have three test log files. Use for loop to simulate (nearly) streaming situation.

In the result folder, there are _main_*.csv files and *.log_*.csv files. The _main_*.csv files will keep appending the new coming log when it has been parse.

Plot

We can use graphviz to visualize the tree-structured of the parser.

python plot_tree.py

 

tree

Tests

sh test.sh
Name Stmts Miss Cover
spell/init.py 0 0 100%
spellpy/spell.py 319 188 41%
test/test_spellpu.py 65 1 98%
TOTAL 387 189 51%

TODO

  • Refactor spellpy.py
  • Test coverage

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spellpy-0.0.9.tar.gz (7.6 kB view details)

Uploaded Source

Built Distribution

spellpy-0.0.9-py3-none-any.whl (19.1 kB view details)

Uploaded Python 3

File details

Details for the file spellpy-0.0.9.tar.gz.

File metadata

  • Download URL: spellpy-0.0.9.tar.gz
  • Upload date:
  • Size: 7.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.4.2 requests/2.21.0 setuptools/46.0.0 requests-toolbelt/0.9.1 tqdm/4.28.1 CPython/3.7.1

File hashes

Hashes for spellpy-0.0.9.tar.gz
Algorithm Hash digest
SHA256 ce9bdf57bdd6d078ecea4da8e0b898b598df09238170bcfa325ca60ac5ecea5f
MD5 7d51504c532e249a1f70811c279d686f
BLAKE2b-256 c95b367a7124d91e4b7abc308a4a555f0640deb13fcbabf761bd9e1e51738742

See more details on using hashes here.

File details

Details for the file spellpy-0.0.9-py3-none-any.whl.

File metadata

  • Download URL: spellpy-0.0.9-py3-none-any.whl
  • Upload date:
  • Size: 19.1 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.4.2 requests/2.21.0 setuptools/46.0.0 requests-toolbelt/0.9.1 tqdm/4.28.1 CPython/3.7.1

File hashes

Hashes for spellpy-0.0.9-py3-none-any.whl
Algorithm Hash digest
SHA256 915044edb21633c0c73a150ed7a27f8ed8e09bd2efbc920fba5d3bb5110d57e3
MD5 5757fd794a0c7133846e30c9a033247e
BLAKE2b-256 c88af9962aefb51b251cfe42e07d2bebf7cd639acb4e35ebbe26990f5536cf0f

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page