ZLogging - Bro/Zeek logging framework for Python
Online documentation is available at https://zlogging.readthedocs.io/
The ZLogging module provides an easy-to-use bridge between the logging framework of the well-known Bro/Zeek Network Security Monitor (IDS).
As of version 3.0, the Bro project has been officially renamed to Zeek. [1]
It was originally developed and derived from the BroAPT project, which is an APT detection framework based on the Bro/Zeek IDS and extended with highly customised and customisable Python wrappers.
Installation
pip install zloggingUsage
Currently ZLogging supports the two builtin formats as supported by the Bro/Zeek logging framework, i.e. ASCII and JSON.
A typical ASCII log file would be like:
#separator \x09 #set_separator , #empty_field (empty) #unset_field - #path http #open 2020-02-09-18-54-09 #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p trans_depth method host uri referrer version user_agent origin request_body_len response_body_len status_code status_msg info_code info_msg tags username password proxied orig_fuids orig_filenames orig_mime_types resp_fuids resp_filenames resp_mime_types #types time string addr port addr port count string string string string string string string count count count string count string set[enum] string string set[string] vector[string] vector[string] vector[string] vector[string] vector[string] vector[string] 1581245648.761106 CSksID3S6ZxplpvmXg 192.168.2.108 56475 151.139.128.14 80 1 GET ocsp.sectigo.com /MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFEML0g5PE3oabJGPJOXafjJNRzPIBBSNjF7EVK2K4Xfpm/mbBeG4AY1h4QIQfdsAWJ+CXcbhDVFyNWosjQ== - 1.1 com.apple.trustd/2.0 - 0 471 200 OK - - (empty) - - - - - - FPtlyEAhcf8orBPu7 - application/ocsp-response 1581245651.379048 CuvUnl4HyhQbCs4tXe 192.168.2.108 56483 23.59.247.10 80 1 GET isrg.trustid.ocsp.identrust.com /MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA== - 1.1 com.apple.trustd/2.0 - 0 1398 200 OK - - (empty) - - - - - - FRfFoq3hSZkdCNDf9l - application/ocsp-response 1581245654.396334 CWo4pd1z97XLB2o0h2 192.168.2.108 56486 23.59.247.122 80 1 GET isrg.trustid.ocsp.identrust.com /MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA== - 1.1 com.apple.trustd/2.0 - 0 1398 200 OK - - (empty) - - - - - - FvQehf1pRsGmwDUzJe - application/ocsp-response 1581245692.728840 CxFQzh2ePtsnQhFNX3 192.168.2.108 56527 23.59.247.10 80 1 GET isrg.trustid.ocsp.identrust.com /MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA== - 1.1 com.apple.trustd/2.0 - 0 1398 200 OK - - (empty) - - - - - - FIeFj8WWNyhA1psGg - application/ocsp-response 1581245701.693971 CPZSNk1Y6kDvAN0KZ8 192.168.2.108 56534 23.59.247.122 80 1 GET isrg.trustid.ocsp.identrust.com /MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA== - 1.1 com.apple.trustd/2.0 - 0 1398 200 OK - - (empty) - - - - - - F0fGHe4RPuNBhYWNv6 - application/ocsp-response 1581245707.848088 Cnab6CHFOprdppKi5 192.168.2.108 56542 23.59.247.122 80 1 GET isrg.trustid.ocsp.identrust.com /MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA== - 1.1 com.apple.trustd/2.0 - 0 1398 200 OK - - (empty) - - - - - - FgDBep1h7EPHC8qQB6 - application/ocsp-response 1581245952.784242 CPNd6t3ofePpdNjErl 192.168.2.108 56821 176.31.225.118 80 1 GET tracker.trackerfix.com /announce?info_hash=y\x82es"\x1dV\xde|m\xbe"\xe5\xef\xbe\x04\xb3\x1fW\xfc&peer_id=-qB4210-0ZOn5Ifyl*WF&port=63108&uploaded=0&downloaded=0&left=3225455594&corrupt=0&key=6B23B036&event=started&numwant=200&compact=1&no_peer_id=1&supportcrypto=1&redundant=0 - 1.1 - - 0 0 307 Temporary Redirect - - (empty) - - - - - - - - - 1581245960.123295 CfAkwf2CFI13b24gqf 192.168.2.108 56889 176.31.225.118 80 1 GET tracker.trackerfix.com /announce?info_hash=!u7\xdad\x94x\xecS\x80\x89\x04\x9c\x13#\x84M\x1b\xcd\x1a&peer_id=-qB4210-i36iloGe*QT9&port=63108&uploaded=0&downloaded=0&left=1637966572&corrupt=0&key=ECE6637E&event=started&numwant=200&compact=1&no_peer_id=1&supportcrypto=1&redundant=0 - 1.1 - - 0 0 307 Temporary Redirect - - (empty) - - - - - - - - - #close 2020-02-09-19-01-40
Its corresponding JSON log file would be like:
{"ts": 1581245648.761106, "uid": "CSksID3S6ZxplpvmXg", "id.orig_h": "192.168.2.108", "id.orig_p": 56475, "id.resp_h": "151.139.128.14", "id.resp_p": 80, "trans_depth": 1, "method": "GET", "host": "ocsp.sectigo.com", "uri": "/MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFEML0g5PE3oabJGPJOXafjJNRzPIBBSNjF7EVK2K4Xfpm/mbBeG4AY1h4QIQfdsAWJ+CXcbhDVFyNWosjQ==", "referrer": "-", "version": "1.1", "user_agent": "com.apple.trustd/2.0", "origin": "-", "request_body_len": 0, "response_body_len": 471, "status_code": 200, "status_msg": "OK", "info_code": null, "info_msg": "-", "tags": [], "username": "-", "password": "-", "proxied": null, "orig_fuids": null, "orig_filenames": null, "orig_mime_types": null, "resp_fuids": ["FPtlyEAhcf8orBPu7"], "resp_filenames": null, "resp_mime_types": ["application/ocsp-response"]}
{"ts": 1581245651.379048, "uid": "CuvUnl4HyhQbCs4tXe", "id.orig_h": "192.168.2.108", "id.orig_p": 56483, "id.resp_h": "23.59.247.10", "id.resp_p": 80, "trans_depth": 1, "method": "GET", "host": "isrg.trustid.ocsp.identrust.com", "uri": "/MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA==", "referrer": "-", "version": "1.1", "user_agent": "com.apple.trustd/2.0", "origin": "-", "request_body_len": 0, "response_body_len": 1398, "status_code": 200, "status_msg": "OK", "info_code": null, "info_msg": "-", "tags": [], "username": "-", "password": "-", "proxied": null, "orig_fuids": null, "orig_filenames": null, "orig_mime_types": null, "resp_fuids": ["FRfFoq3hSZkdCNDf9l"], "resp_filenames": null, "resp_mime_types": ["application/ocsp-response"]}
{"ts": 1581245654.396334, "uid": "CWo4pd1z97XLB2o0h2", "id.orig_h": "192.168.2.108", "id.orig_p": 56486, "id.resp_h": "23.59.247.122", "id.resp_p": 80, "trans_depth": 1, "method": "GET", "host": "isrg.trustid.ocsp.identrust.com", "uri": "/MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA==", "referrer": "-", "version": "1.1", "user_agent": "com.apple.trustd/2.0", "origin": "-", "request_body_len": 0, "response_body_len": 1398, "status_code": 200, "status_msg": "OK", "info_code": null, "info_msg": "-", "tags": [], "username": "-", "password": "-", "proxied": null, "orig_fuids": null, "orig_filenames": null, "orig_mime_types": null, "resp_fuids": ["FvQehf1pRsGmwDUzJe"], "resp_filenames": null, "resp_mime_types": ["application/ocsp-response"]}
{"ts": 1581245692.72884, "uid": "CxFQzh2ePtsnQhFNX3", "id.orig_h": "192.168.2.108", "id.orig_p": 56527, "id.resp_h": "23.59.247.10", "id.resp_p": 80, "trans_depth": 1, "method": "GET", "host": "isrg.trustid.ocsp.identrust.com", "uri": "/MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA==", "referrer": "-", "version": "1.1", "user_agent": "com.apple.trustd/2.0", "origin": "-", "request_body_len": 0, "response_body_len": 1398, "status_code": 200, "status_msg": "OK", "info_code": null, "info_msg": "-", "tags": [], "username": "-", "password": "-", "proxied": null, "orig_fuids": null, "orig_filenames": null, "orig_mime_types": null, "resp_fuids": ["FIeFj8WWNyhA1psGg"], "resp_filenames": null, "resp_mime_types": ["application/ocsp-response"]}
{"ts": 1581245701.693971, "uid": "CPZSNk1Y6kDvAN0KZ8", "id.orig_h": "192.168.2.108", "id.orig_p": 56534, "id.resp_h": "23.59.247.122", "id.resp_p": 80, "trans_depth": 1, "method": "GET", "host": "isrg.trustid.ocsp.identrust.com", "uri": "/MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA==", "referrer": "-", "version": "1.1", "user_agent": "com.apple.trustd/2.0", "origin": "-", "request_body_len": 0, "response_body_len": 1398, "status_code": 200, "status_msg": "OK", "info_code": null, "info_msg": "-", "tags": [], "username": "-", "password": "-", "proxied": null, "orig_fuids": null, "orig_filenames": null, "orig_mime_types": null, "resp_fuids": ["F0fGHe4RPuNBhYWNv6"], "resp_filenames": null, "resp_mime_types": ["application/ocsp-response"]}
{"ts": 1581245707.848088, "uid": "Cnab6CHFOprdppKi5", "id.orig_h": "192.168.2.108", "id.orig_p": 56542, "id.resp_h": "23.59.247.122", "id.resp_p": 80, "trans_depth": 1, "method": "GET", "host": "isrg.trustid.ocsp.identrust.com", "uri": "/MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA==", "referrer": "-", "version": "1.1", "user_agent": "com.apple.trustd/2.0", "origin": "-", "request_body_len": 0, "response_body_len": 1398, "status_code": 200, "status_msg": "OK", "info_code": null, "info_msg": "-", "tags": [], "username": "-", "password": "-", "proxied": null, "orig_fuids": null, "orig_filenames": null, "orig_mime_types": null, "resp_fuids": ["FgDBep1h7EPHC8qQB6"], "resp_filenames": null, "resp_mime_types": ["application/ocsp-response"]}
{"ts": 1581245952.784242, "uid": "CPNd6t3ofePpdNjErl", "id.orig_h": "192.168.2.108", "id.orig_p": 56821, "id.resp_h": "176.31.225.118", "id.resp_p": 80, "trans_depth": 1, "method": "GET", "host": "tracker.trackerfix.com", "uri": "/announce?info_hash=y\\x82es\"\\x1dV\\xde|m\\xbe\"\\xe5\\xef\\xbe\\x04\\xb3\\x1fW\\xfc&peer_id=-qB4210-0ZOn5Ifyl*WF&port=63108&uploaded=0&downloaded=0&left=3225455594&corrupt=0&key=6B23B036&event=started&numwant=200&compact=1&no_peer_id=1&supportcrypto=1&redundant=0", "referrer": "-", "version": "1.1", "user_agent": "-", "origin": "-", "request_body_len": 0, "response_body_len": 0, "status_code": 307, "status_msg": "Temporary Redirect", "info_code": null, "info_msg": "-", "tags": [], "username": "-", "password": "-", "proxied": null, "orig_fuids": null, "orig_filenames": null, "orig_mime_types": null, "resp_fuids": null, "resp_filenames": null, "resp_mime_types": null}
{"ts": 1581245960.123295, "uid": "CfAkwf2CFI13b24gqf", "id.orig_h": "192.168.2.108", "id.orig_p": 56889, "id.resp_h": "176.31.225.118", "id.resp_p": 80, "trans_depth": 1, "method": "GET", "host": "tracker.trackerfix.com", "uri": "/announce?info_hash=!u7\\xdad\\x94x\\xecS\\x80\\x89\\x04\\x9c\\x13#\\x84M\\x1b\\xcd\\x1a&peer_id=-qB4210-i36iloGe*QT9&port=63108&uploaded=0&downloaded=0&left=1637966572&corrupt=0&key=ECE6637E&event=started&numwant=200&compact=1&no_peer_id=1&supportcrypto=1&redundant=0", "referrer": "-", "version": "1.1", "user_agent": "-", "origin": "-", "request_body_len": 0, "response_body_len": 0, "status_code": 307, "status_msg": "Temporary Redirect", "info_code": null, "info_msg": "-", "tags": [], "username": "-", "password": "-", "proxied": null, "orig_fuids": null, "orig_filenames": null, "orig_mime_types": null, "resp_fuids": null, "resp_filenames": null, "resp_mime_types": null}
How to Load/Parse a Log File?
To load (parse) a log file generically, i.e. when you don’t know what format the log file is, you can simple call the ~zlogging.loader.parse, zlogging.loader.loads functions:
# to parse log at filename
>>> parse('path/to/log')
# to load log from a file object
>>> with open('path/to/log', 'rb') as file:
... load(file)
# to load log from a string
>>> with open('/path/to/log', 'rb') as file:
... loads(file.read())
If you do know the format, you may call the specified functions for each zlogging.loader.parse_ascii and zlogging.loader.parse_json, etc.
If you would like to customise your own parser, just subclass zlogging.loader.BaseParser and implement your own ideas.
How to Dump/Write a Log File?
Before dumping (writing) a log file, you need to create a log data model first. Just like in the Bro/Zeek script language, when customise logging, you need to notify the logging framework with a new log stream. Here, in ZLogging, we introduced data model for the same purpose.
A data model is a subclass of zlogging.model.Model with fields and data types declared. A typical data model can be as following:
class MyLog(Model):
field_one = StringType()
field_two = SetType(element_type=PortType)
where field_one is string type, i.e. zlogging.types.StringType; and field_two is set[port] types, i.e. zlogging.types.SetType of zlogging.types.PortType.
Or you may use type annotations as PEP 484 introduced when declaring data models. All available type hints can be found in zlogging.typing:
class MyLog(Model):
field_one: zeek_string
field_two: zeek_set[zeek_port]
After declaration of your data model, you can know dump (write) your log file with the corresponding functions.
If you would like to customise your own writer, just subclass zlogging.loader.BaseWriter and implement your own ideas.
1 maintainer
Metadata
1 maintainer
Release files for zlogging 0.1.3.post11
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| zlogging-0.1.3.post11.tar.gz | 61.3 kB | Without provenance | Details for zlogging-0.1.3.post11.tar.gz |
Built distributions (wheels)
| File | Actions Reset | |||
|---|---|---|---|---|
| zlogging-0.1.3.post11-cp314-none-any.whl | CPython 3.14 | none | any | Details for zlogging-0.1.3.post11-cp314-none-any.whl |
| zlogging-0.1.3.post11-cp313-none-any.whl | CPython 3.13 | none | any | Details for zlogging-0.1.3.post11-cp313-none-any.whl |
| zlogging-0.1.3.post11-cp312-none-any.whl | CPython 3.12 | none | any | Details for zlogging-0.1.3.post11-cp312-none-any.whl |
| zlogging-0.1.3.post11-cp311-none-any.whl | CPython 3.11 | none | any | Details for zlogging-0.1.3.post11-cp311-none-any.whl |
Total release size: 266.4 kB
Release files / zlogging-0.1.3.post11.tar.gz
| Download URL | zlogging-0.1.3.post11.tar.gz |
|---|---|
| Size | 61.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ad381899e252a900f5a27ba6fa32f23a16f57d8cf885690cdec7195f8617e060
|
|
BLAKE2b-256 checksum How to use checksums |
3b8a283fa3b331417d43395d058dd6546a524b18c55320ebcf57ac97a3abf2a3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / zlogging-0.1.3.post11-cp314-none-any.whl
| Download URL | zlogging-0.1.3.post11-cp314-none-any.whl |
|---|---|
| Size | 51.3 kB |
| Tags | CPython 3.14 |
|
SHA-256 checksum How to use checksums |
a2dbbdbc9f3062489744abf0b1c1d7ec29009f6e1144900c4ee6621068d3aa70
|
|
BLAKE2b-256 checksum How to use checksums |
46bbd058e894a2e43907b163da07c84e16facd081f985bff161b94402c593068
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / zlogging-0.1.3.post11-cp313-none-any.whl
| Download URL | zlogging-0.1.3.post11-cp313-none-any.whl |
|---|---|
| Size | 51.3 kB |
| Tags | CPython 3.13 |
|
SHA-256 checksum How to use checksums |
4a59bb58c8c89c1df13f2ae49a4a9980871c17d834774cfb9a011619d80716d4
|
|
BLAKE2b-256 checksum How to use checksums |
e4c7e8562651debe2f5bb46367b3c1319f227742fb8af8c66795bae992aa1fbc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / zlogging-0.1.3.post11-cp312-none-any.whl
| Download URL | zlogging-0.1.3.post11-cp312-none-any.whl |
|---|---|
| Size | 51.3 kB |
| Tags | CPython 3.12 |
|
SHA-256 checksum How to use checksums |
e33aba3eba8d49e2317e97dd869908a0ffeb8296746837a1c2c7334b954f94c7
|
|
BLAKE2b-256 checksum How to use checksums |
9afa8ead674a8c341c0595633d58fa7dfd5cd672d924140eafb6331d8c68298f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / zlogging-0.1.3.post11-cp311-none-any.whl
| Download URL | zlogging-0.1.3.post11-cp311-none-any.whl |
|---|---|
| Size | 51.3 kB |
| Tags | CPython 3.11 |
|
SHA-256 checksum How to use checksums |
64c596df1fc736ba323036b78b14619133b4a248e0cc8f6c5007738a39a05b6d
|
|
BLAKE2b-256 checksum How to use checksums |
4417bbdfe01f23f9f676f7420497567d68209f0c160ba180cdeb6dce4bed7afe
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Security for zlogging 0.1.3.post11
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
