Spell
Use python implement the paper Spell: Streaming Parsing of System Event Logs from Min Du, Feifei Li @University of Utah.
This implement is refactored and enhancement version of logpai's logparser.
Install
pip install spellpy
Getting Started
python example.py
After executing the line above, the result folder will be created and you will see two files: structured.csv and templates.csv.
*_main_structured.csv
| ... | Level | Component | Content | EventId | EventTemplate | ParameterList |
|---|---|---|---|---|---|---|
| ... | INFO | dfs.DataNode$DataXceiver | Receiving block blk_-1608999687919862906 src: /10.250.19.102:54106 dest: /10.250.19.102:50010 | f57d69cf | Receiving block blk_-1608999687919862906 src <*> <*> dest <*> 50010 | ['/10.250.19.102:54106', '/10.250.19.102'] |
| ... | INFO | dfs.DataNode$PacketResponder | PacketResponder 1 for block blk_-1608999687919862906 terminating | 7b619377 | PacketResponder <*> for block blk_-1608999687919862906 terminating | ['1'] |
| ... | INFO | dfs.DataNode$DataXceiver | Receiving block blk_-1608999687919862906 src: /10.250.10.6:40524 dest: /10.250.10.6:50010 | f57d69cf | Receiving block blk_-1608999687919862906 src <*> <*> dest <*> 50010 | ['/10.250.10.6:40524', '/10.250.10.6'] |
*_main_templates.csv
| EventId | EventTemplate | Occurrences |
|---|---|---|
| 6af214fd | Receiving block <*> src <*> <*> dest <*> 50010 | 5 |
| 26ae4ce0 | BLOCK* NameSystem.allocateBlock <*> | 2 |
| dc2c74b7 | PacketResponder <*> for block <*> terminating | 4 |
As you see, there have three test log files. Use for loop to simulate (nearly) streaming situation.
In the result folder, there are _main_*.csv files and *.log_*.csv files. The _main_*.csv files will keep appending the new coming log when it has been parse.
Plot
We can use graphviz to visualize the tree-structured of the parser.
python plot_tree.py
Tests
sh test.sh
| Name | Stmts | Miss | Cover |
|---|---|---|---|
| spell/init.py | 0 | 0 | 100% |
| spellpy/spell.py | 319 | 188 | 41% |
| test/test_spellpu.py | 65 | 1 | 98% |
| TOTAL | 387 | 189 | 51% |
TODO
- Refactor
spellpy.py - Test coverage
1 maintainer
Metadata
1 maintainer
Release files for spellpy 0.0.9
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| spellpy-0.0.9.tar.gz | 7.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| spellpy-0.0.9-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 26.7 kB
Release files / spellpy-0.0.9.tar.gz
| Download URL | spellpy-0.0.9.tar.gz |
|---|---|
| Size | 7.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ce9bdf57bdd6d078ecea4da8e0b898b598df09238170bcfa325ca60ac5ecea5f
|
|
BLAKE2b-256 checksum How to use checksums |
c95b367a7124d91e4b7abc308a4a555f0640deb13fcbabf761bd9e1e51738742
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.4.2 requests/2.21.0 setuptools/46.0.0 requests-toolbelt/0.9.1 tqdm/4.28.1 CPython/3.7.1
|
Release files / spellpy-0.0.9-py3-none-any.whl
| Download URL | spellpy-0.0.9-py3-none-any.whl |
|---|---|
| Size | 19.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
915044edb21633c0c73a150ed7a27f8ed8e09bd2efbc920fba5d3bb5110d57e3
|
|
BLAKE2b-256 checksum How to use checksums |
c88af9962aefb51b251cfe42e07d2bebf7cd639acb4e35ebbe26990f5536cf0f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.4.2 requests/2.21.0 setuptools/46.0.0 requests-toolbelt/0.9.1 tqdm/4.28.1 CPython/3.7.1
|
