Skip to main content

Online documentation is available at https://zlogging.readthedocs.io/

The ZLogging module provides an easy-to-use bridge between the logging framework of the well-known Bro/Zeek Network Security Monitor (IDS).

As of version 3.0, the Bro project has been officially renamed to Zeek. [1]

It was originally developed and derived from the BroAPT project, which is an APT detection framework based on the Bro/Zeek IDS and extended with highly customised and customisable Python wrappers.

Installation

pip install zlogging

Usage

Currently ZLogging supports the two builtin formats as supported by the Bro/Zeek logging framework, i.e. ASCII and JSON.

A typical ASCII log file would be like:

#separator \x09
#set_separator      ,
#empty_field        (empty)
#unset_field        -
#path       http
#open       2020-02-09-18-54-09
#fields     ts      uid     id.orig_h       id.orig_p       id.resp_h       id.resp_p       trans_depth     method  host    uri     referrer        version user_agent      origin  request_body_len        response_body_len       status_code     status_msg      info_code       info_msg        tags    username        password        proxied orig_fuids      orig_filenames  orig_mime_types resp_fuids      resp_filenames  resp_mime_types
#types      time    string  addr    port    addr    port    count   string  string  string  string  string  string  string  count   count   count   string  count   string  set[enum]       string  string  set[string]     vector[string]  vector[string]  vector[string]  vector[string]  vector[string]  vector[string]
1581245648.761106   CSksID3S6ZxplpvmXg      192.168.2.108   56475   151.139.128.14  80      1       GET     ocsp.sectigo.com        /MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFEML0g5PE3oabJGPJOXafjJNRzPIBBSNjF7EVK2K4Xfpm/mbBeG4AY1h4QIQfdsAWJ+CXcbhDVFyNWosjQ==       -       1.1     com.apple.trustd/2.0    -       0       471     200     OK      -       -       (empty) -       -       -       -       -       -       FPtlyEAhcf8orBPu7       -       application/ocsp-response
1581245651.379048   CuvUnl4HyhQbCs4tXe      192.168.2.108   56483   23.59.247.10    80      1       GET     isrg.trustid.ocsp.identrust.com /MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA==       -       1.1     com.apple.trustd/2.0    -       0       1398    200     OK      -       -       (empty) -       -       -       -       -       -       FRfFoq3hSZkdCNDf9l      -       application/ocsp-response
1581245654.396334   CWo4pd1z97XLB2o0h2      192.168.2.108   56486   23.59.247.122   80      1       GET     isrg.trustid.ocsp.identrust.com /MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA==       -       1.1     com.apple.trustd/2.0    -       0       1398    200     OK      -       -       (empty) -       -       -       -       -       -       FvQehf1pRsGmwDUzJe      -       application/ocsp-response
1581245692.728840   CxFQzh2ePtsnQhFNX3      192.168.2.108   56527   23.59.247.10    80      1       GET     isrg.trustid.ocsp.identrust.com /MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA==       -       1.1     com.apple.trustd/2.0    -       0       1398    200     OK      -       -       (empty) -       -       -       -       -       -       FIeFj8WWNyhA1psGg       -       application/ocsp-response
1581245701.693971   CPZSNk1Y6kDvAN0KZ8      192.168.2.108   56534   23.59.247.122   80      1       GET     isrg.trustid.ocsp.identrust.com /MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA==       -       1.1     com.apple.trustd/2.0    -       0       1398    200     OK      -       -       (empty) -       -       -       -       -       -       F0fGHe4RPuNBhYWNv6      -       application/ocsp-response
1581245707.848088   Cnab6CHFOprdppKi5       192.168.2.108   56542   23.59.247.122   80      1       GET     isrg.trustid.ocsp.identrust.com /MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA==       -       1.1     com.apple.trustd/2.0    -       0       1398    200     OK      -       -       (empty) -       -       -       -       -       -       FgDBep1h7EPHC8qQB6      -       application/ocsp-response
1581245952.784242   CPNd6t3ofePpdNjErl      192.168.2.108   56821   176.31.225.118  80      1       GET     tracker.trackerfix.com  /announce?info_hash=y\x82es"\x1dV\xde|m\xbe"\xe5\xef\xbe\x04\xb3\x1fW\xfc&peer_id=-qB4210-0ZOn5Ifyl*WF&port=63108&uploaded=0&downloaded=0&left=3225455594&corrupt=0&key=6B23B036&event=started&numwant=200&compact=1&no_peer_id=1&supportcrypto=1&redundant=0   -       1.1     -       -       0       0       307     Temporary Redirect      -       -       (empty) -       -       -       -       -       -       -       -       -
1581245960.123295   CfAkwf2CFI13b24gqf      192.168.2.108   56889   176.31.225.118  80      1       GET     tracker.trackerfix.com  /announce?info_hash=!u7\xdad\x94x\xecS\x80\x89\x04\x9c\x13#\x84M\x1b\xcd\x1a&peer_id=-qB4210-i36iloGe*QT9&port=63108&uploaded=0&downloaded=0&left=1637966572&corrupt=0&key=ECE6637E&event=started&numwant=200&compact=1&no_peer_id=1&supportcrypto=1&redundant=0        -       1.1     -       -       0       0       307     Temporary Redirect      -       -       (empty) -       -       -       -       -       -       -       -       -
#close      2020-02-09-19-01-40

Its corresponding JSON log file would be like:

{"ts": 1581245648.761106, "uid": "CSksID3S6ZxplpvmXg", "id.orig_h": "192.168.2.108", "id.orig_p": 56475, "id.resp_h": "151.139.128.14", "id.resp_p": 80, "trans_depth": 1, "method": "GET", "host": "ocsp.sectigo.com", "uri": "/MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFEML0g5PE3oabJGPJOXafjJNRzPIBBSNjF7EVK2K4Xfpm/mbBeG4AY1h4QIQfdsAWJ+CXcbhDVFyNWosjQ==", "referrer": "-", "version": "1.1", "user_agent": "com.apple.trustd/2.0", "origin": "-", "request_body_len": 0, "response_body_len": 471, "status_code": 200, "status_msg": "OK", "info_code": null, "info_msg": "-", "tags": [], "username": "-", "password": "-", "proxied": null, "orig_fuids": null, "orig_filenames": null, "orig_mime_types": null, "resp_fuids": ["FPtlyEAhcf8orBPu7"], "resp_filenames": null, "resp_mime_types": ["application/ocsp-response"]}
{"ts": 1581245651.379048, "uid": "CuvUnl4HyhQbCs4tXe", "id.orig_h": "192.168.2.108", "id.orig_p": 56483, "id.resp_h": "23.59.247.10", "id.resp_p": 80, "trans_depth": 1, "method": "GET", "host": "isrg.trustid.ocsp.identrust.com", "uri": "/MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA==", "referrer": "-", "version": "1.1", "user_agent": "com.apple.trustd/2.0", "origin": "-", "request_body_len": 0, "response_body_len": 1398, "status_code": 200, "status_msg": "OK", "info_code": null, "info_msg": "-", "tags": [], "username": "-", "password": "-", "proxied": null, "orig_fuids": null, "orig_filenames": null, "orig_mime_types": null, "resp_fuids": ["FRfFoq3hSZkdCNDf9l"], "resp_filenames": null, "resp_mime_types": ["application/ocsp-response"]}
{"ts": 1581245654.396334, "uid": "CWo4pd1z97XLB2o0h2", "id.orig_h": "192.168.2.108", "id.orig_p": 56486, "id.resp_h": "23.59.247.122", "id.resp_p": 80, "trans_depth": 1, "method": "GET", "host": "isrg.trustid.ocsp.identrust.com", "uri": "/MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA==", "referrer": "-", "version": "1.1", "user_agent": "com.apple.trustd/2.0", "origin": "-", "request_body_len": 0, "response_body_len": 1398, "status_code": 200, "status_msg": "OK", "info_code": null, "info_msg": "-", "tags": [], "username": "-", "password": "-", "proxied": null, "orig_fuids": null, "orig_filenames": null, "orig_mime_types": null, "resp_fuids": ["FvQehf1pRsGmwDUzJe"], "resp_filenames": null, "resp_mime_types": ["application/ocsp-response"]}
{"ts": 1581245692.72884, "uid": "CxFQzh2ePtsnQhFNX3", "id.orig_h": "192.168.2.108", "id.orig_p": 56527, "id.resp_h": "23.59.247.10", "id.resp_p": 80, "trans_depth": 1, "method": "GET", "host": "isrg.trustid.ocsp.identrust.com", "uri": "/MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA==", "referrer": "-", "version": "1.1", "user_agent": "com.apple.trustd/2.0", "origin": "-", "request_body_len": 0, "response_body_len": 1398, "status_code": 200, "status_msg": "OK", "info_code": null, "info_msg": "-", "tags": [], "username": "-", "password": "-", "proxied": null, "orig_fuids": null, "orig_filenames": null, "orig_mime_types": null, "resp_fuids": ["FIeFj8WWNyhA1psGg"], "resp_filenames": null, "resp_mime_types": ["application/ocsp-response"]}
{"ts": 1581245701.693971, "uid": "CPZSNk1Y6kDvAN0KZ8", "id.orig_h": "192.168.2.108", "id.orig_p": 56534, "id.resp_h": "23.59.247.122", "id.resp_p": 80, "trans_depth": 1, "method": "GET", "host": "isrg.trustid.ocsp.identrust.com", "uri": "/MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA==", "referrer": "-", "version": "1.1", "user_agent": "com.apple.trustd/2.0", "origin": "-", "request_body_len": 0, "response_body_len": 1398, "status_code": 200, "status_msg": "OK", "info_code": null, "info_msg": "-", "tags": [], "username": "-", "password": "-", "proxied": null, "orig_fuids": null, "orig_filenames": null, "orig_mime_types": null, "resp_fuids": ["F0fGHe4RPuNBhYWNv6"], "resp_filenames": null, "resp_mime_types": ["application/ocsp-response"]}
{"ts": 1581245707.848088, "uid": "Cnab6CHFOprdppKi5", "id.orig_h": "192.168.2.108", "id.orig_p": 56542, "id.resp_h": "23.59.247.122", "id.resp_p": 80, "trans_depth": 1, "method": "GET", "host": "isrg.trustid.ocsp.identrust.com", "uri": "/MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFG/0aE1DEtJIYoGcwCs9Rywdii+mBBTEp7Gkeyxx+tvhS5B1/8QVYIWJEAIQCgFBQgAAAVOFc2oLheynCA==", "referrer": "-", "version": "1.1", "user_agent": "com.apple.trustd/2.0", "origin": "-", "request_body_len": 0, "response_body_len": 1398, "status_code": 200, "status_msg": "OK", "info_code": null, "info_msg": "-", "tags": [], "username": "-", "password": "-", "proxied": null, "orig_fuids": null, "orig_filenames": null, "orig_mime_types": null, "resp_fuids": ["FgDBep1h7EPHC8qQB6"], "resp_filenames": null, "resp_mime_types": ["application/ocsp-response"]}
{"ts": 1581245952.784242, "uid": "CPNd6t3ofePpdNjErl", "id.orig_h": "192.168.2.108", "id.orig_p": 56821, "id.resp_h": "176.31.225.118", "id.resp_p": 80, "trans_depth": 1, "method": "GET", "host": "tracker.trackerfix.com", "uri": "/announce?info_hash=y\\x82es\"\\x1dV\\xde|m\\xbe\"\\xe5\\xef\\xbe\\x04\\xb3\\x1fW\\xfc&peer_id=-qB4210-0ZOn5Ifyl*WF&port=63108&uploaded=0&downloaded=0&left=3225455594&corrupt=0&key=6B23B036&event=started&numwant=200&compact=1&no_peer_id=1&supportcrypto=1&redundant=0", "referrer": "-", "version": "1.1", "user_agent": "-", "origin": "-", "request_body_len": 0, "response_body_len": 0, "status_code": 307, "status_msg": "Temporary Redirect", "info_code": null, "info_msg": "-", "tags": [], "username": "-", "password": "-", "proxied": null, "orig_fuids": null, "orig_filenames": null, "orig_mime_types": null, "resp_fuids": null, "resp_filenames": null, "resp_mime_types": null}
{"ts": 1581245960.123295, "uid": "CfAkwf2CFI13b24gqf", "id.orig_h": "192.168.2.108", "id.orig_p": 56889, "id.resp_h": "176.31.225.118", "id.resp_p": 80, "trans_depth": 1, "method": "GET", "host": "tracker.trackerfix.com", "uri": "/announce?info_hash=!u7\\xdad\\x94x\\xecS\\x80\\x89\\x04\\x9c\\x13#\\x84M\\x1b\\xcd\\x1a&peer_id=-qB4210-i36iloGe*QT9&port=63108&uploaded=0&downloaded=0&left=1637966572&corrupt=0&key=ECE6637E&event=started&numwant=200&compact=1&no_peer_id=1&supportcrypto=1&redundant=0", "referrer": "-", "version": "1.1", "user_agent": "-", "origin": "-", "request_body_len": 0, "response_body_len": 0, "status_code": 307, "status_msg": "Temporary Redirect", "info_code": null, "info_msg": "-", "tags": [], "username": "-", "password": "-", "proxied": null, "orig_fuids": null, "orig_filenames": null, "orig_mime_types": null, "resp_fuids": null, "resp_filenames": null, "resp_mime_types": null}

How to Load/Parse a Log File?

To load (parse) a log file generically, i.e. when you don’t know what format the log file is, you can simple call the ~zlogging.loader.parse, zlogging.loader.loads functions:

# to parse log at filename
>>> parse('path/to/log')
# to load log from a file object
>>> with open('path/to/log', 'rb') as file:
...     load(file)
# to load log from a string
>>> with open('/path/to/log', 'rb') as file:
...     loads(file.read())

If you do know the format, you may call the specified functions for each zlogging.loader.parse_ascii and zlogging.loader.parse_json, etc.

If you would like to customise your own parser, just subclass zlogging.loader.BaseParser and implement your own ideas.

How to Dump/Write a Log File?

Before dumping (writing) a log file, you need to create a log data model first. Just like in the Bro/Zeek script language, when customise logging, you need to notify the logging framework with a new log stream. Here, in ZLogging, we introduced data model for the same purpose.

A data model is a subclass of zlogging.model.Model with fields and data types declared. A typical data model can be as following:

class MyLog(Model):
    field_one = StringType()
    field_two = SetType(element_type=PortType)

where field_one is string type, i.e. zlogging.types.StringType; and field_two is set[port] types, i.e. zlogging.types.SetType of zlogging.types.PortType.

Or you may use type annotations as PEP 484 introduced when declaring data models. All available type hints can be found in zlogging.typing:

class MyLog(Model):
    field_one: zeek_string
    field_two: zeek_set[zeek_port]

After declaration of your data model, you can know dump (write) your log file with the corresponding functions.

If you would like to customise your own writer, just subclass zlogging.loader.BaseWriter and implement your own ideas.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

zlogging-0.1.3.post10.tar.gz (89.3 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

zlogging-0.1.3.post10-cp314-none-any.whl (92.0 kB view details)

Uploaded CPython 3.14

zlogging-0.1.3.post10-cp313-none-any.whl (92.0 kB view details)

Uploaded CPython 3.13

zlogging-0.1.3.post10-cp311-none-any.whl (92.0 kB view details)

Uploaded CPython 3.11

zlogging-0.1.3.post10-cp310-none-any.whl (92.0 kB view details)

Uploaded CPython 3.10

File details

Details for the file zlogging-0.1.3.post10.tar.gz.

File metadata

  • Download URL: zlogging-0.1.3.post10.tar.gz
  • Upload date:
  • Size: 89.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for zlogging-0.1.3.post10.tar.gz
Algorithm Hash digest
SHA256 6daf0c8df9d43f85e82c3f87bfc6c1e69e4d5723a2dbe6da33a1aa79925a7d81
MD5 dd5b5cd4249c989c3cc5ee231aa24784
BLAKE2b-256 cc4bab5090d1653f3c1e54a4c694ecddd2fd95b029a3cbe919c62480e02ac2e6

See more details on using hashes here.

File details

Details for the file zlogging-0.1.3.post10-cp314-none-any.whl.

File metadata

File hashes

Hashes for zlogging-0.1.3.post10-cp314-none-any.whl
Algorithm Hash digest
SHA256 87ad5868ac29620ccc3c3bb124c796e328bd2d5722d9997f13642f760c90ee78
MD5 381af633299a300eb1dcbd82c0ddc5bb
BLAKE2b-256 f765013af33a918318c516d6cbc31e1ecb830015e8d1f96aa610fc3c26fb3908

See more details on using hashes here.

File details

Details for the file zlogging-0.1.3.post10-cp313-none-any.whl.

File metadata

File hashes

Hashes for zlogging-0.1.3.post10-cp313-none-any.whl
Algorithm Hash digest
SHA256 f5bd26c6c7488a973c138a32b8f0f188ec45c46fc3a414219c0f6a2551a83ece
MD5 58b229af8f8aa234f457bfc193acd1a0
BLAKE2b-256 b53aea1816b0145494dd94a8348ef43aaff473d372ae78656f70f88165f86729

See more details on using hashes here.

File details

Details for the file zlogging-0.1.3.post10-cp311-none-any.whl.

File metadata

File hashes

Hashes for zlogging-0.1.3.post10-cp311-none-any.whl
Algorithm Hash digest
SHA256 c111cddcab54e76c70612e5d9cc48b4962f364133099370a13aa27365e1d4cef
MD5 5f5b300024a42e3f4528e2d47a17e0b8
BLAKE2b-256 fd0d8abe3bdeb25704d44d739aeeebc8f8eae34bab2548bef9212aab9ad4a784

See more details on using hashes here.

File details

Details for the file zlogging-0.1.3.post10-cp310-none-any.whl.

File metadata

File hashes

Hashes for zlogging-0.1.3.post10-cp310-none-any.whl
Algorithm Hash digest
SHA256 2afe3a9d8ae697bc46502f9e841663a165a7fcb6f80c856f533ea4f90cc6ee81
MD5 5c74e4e35acd8a2b1fd292e3d0687484
BLAKE2b-256 eeb5736484d731c5feb9503321726cb7e75783f1579496a7f70e3c61b0af33b2

See more details on using hashes here.

Release history Release notifications | RSS feed

This release

0.1.3.post10 This release

5 files

0.1.3.post9

4 files

0.1.3.post8

4 files

0.1.3.post7

4 files

0.1.3.post6

3 files

0.1.3.post5

5 files

0.1.3.post4

9 files

0.1.3.post3

9 files

0.1.3.post2

7 files

0.1.3.post1

7 files

0.1.3

7 files

0.1.2.post5

7 files

0.1.2.post4

7 files

0.1.2.post3

6 files

0.1.2.post2

7 files

0.1.2.post1

2 files

0.1.2

2 files

0.1.1

2 files

0.1.0.post1

2 files

0.1.0

3 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page