gool

Usage

gool is a cli logs clustering tool using drain3 library. It takes one or more log files as input and output clusters of your logs. It is a tactical tool to help for logs investigation, it allows the user to focus on only given logs based on regexp or time. Moreover this tool allows to compare your logs with a baseline (compare logs where your bug happens with the logs when there was nos bug).

The repository also provides gool-merge, a cli tool that merges several log files into one, ordering lines by timestamp. See Merge several log files ordered by time below.

Configuration

The user can tune some settings to build right size clusters and how to filter the input logs :

similarity-threshold : drain3 parameter (default 0.4) between 0 and 1. Value of 1 will lead to cluster with the same exact lines only. The higher, the more clusters you will get. This setting can be tunned in configuration file or on command line.
tree-depth : drain3 parameter (default 4) which high value lead to more clusters. This setting can be tunned in configuration file or on command line.
pattern masking : drain3 parameter to mask pattern so lines can be group easier. For example we can replace all IPs, or times before processing. This setting can only be tunned in configuration file.
filter : filter to only parse some of the line of the input files. For example if the user is focuses on error we can imagine something like : '.*(| Warning || Error ).*'.
time_max : max time of log lines, line with higher timestamp are discarded.
time_max : max time of log lines, line with higher timestamp are discarded.
time_pattern: pattern used to capture timestamp of each log line. Default will be "(\d{2}:\d{2}:\d{2}(?:.\d{1,9})?)". When set, it must be used with time_format.
time_format: Format code (for strptime) of the time extracted of each line to convert string into datetime to allow filtering. By default, it will be "%H:%M:%S.%f" or "%H:%M:%S".
baseline: Generate another cluster as a baseline to compare with it.

By default configuration file is taken from ~/.config/drain3.ini. A default file is generated if there is no.

For more details on drain3 parameters check the official repository : https://github.com/logpai/Drain3.

Launch gool

The examples below use two kind of logs: Zookeeper and Apache. Below some example of log lines:

head tests/data/log/Zookeeper_2k.log

2015-07-29 17:41:44,747 - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@774] - Notification time out: 3200
2015-07-29 19:04:12,394 - INFO  [/10.10.34.11:3888:QuorumCnxManager$Listener@493] - Received connection request /10.10.34.11:45307
2015-07-29 19:04:29,071 - WARN  [SendWorker:188978561024:QuorumCnxManager$SendWorker@688] - Send worker leaving thread
2015-07-29 19:04:29,079 - WARN  [SendWorker:188978561024:QuorumCnxManager$SendWorker@679] - Interrupted while waiting for message on queue
2015-07-29 19:13:17,524 - WARN  [SendWorker:188978561024:QuorumCnxManager$SendWorker@688] - Send worker leaving thread
2015-07-29 19:13:24,282 - WARN  [RecvWorker:188978561024:QuorumCnxManager$RecvWorker@762] - Connection broken for id 188978561024, my id = 1, error =
2015-07-29 19:13:24,370 - INFO  [/10.10.34.11:3888:QuorumCnxManager$Listener@493] - Received connection request /10.10.34.13:57707
2015-07-29 19:13:27,721 - WARN  [RecvWorker:188978561024:QuorumCnxManager$RecvWorker@762] - Connection broken for id 188978561024, my id = 1, error =
2015-07-29 19:13:34,382 - WARN  [SendWorker:188978561024:QuorumCnxManager$SendWorker@679] - Interrupted while waiting for message on queue
2015-07-29 19:13:37,626 - WARN  [SendWorker:188978561024:QuorumCnxManager$SendWorker@688] - Send worker leaving thread

head tests/data/log/Apache_2k.log

[Sun Dec 04 04:47:44 2005] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties
[Sun Dec 04 04:47:44 2005] [error] mod_jk child workerEnv in error state 6
[Sun Dec 04 04:51:08 2005] [notice] jk2_init() Found child 6725 in scoreboard slot 10
[Sun Dec 04 04:51:09 2005] [notice] jk2_init() Found child 6726 in scoreboard slot 8
[Sun Dec 04 04:51:09 2005] [notice] jk2_init() Found child 6728 in scoreboard slot 6
[Sun Dec 04 04:51:14 2005] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties
[Sun Dec 04 04:51:14 2005] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties
[Sun Dec 04 04:51:14 2005] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties
[Sun Dec 04 04:51:18 2005] [error] mod_jk child workerEnv in error state 6
[Sun Dec 04 04:51:18 2005] [error] mod_jk child workerEnv in error state 6

Launch gool and filter lines before processing

gool tests/data/log/Zookeeper_2k.log -f ".*WARN.*"

11:34:48.805482 INFO     log_clustering : Loading configuration from /home/godardo/.drain3.ini                                                       log_clustering.py:111
11:34:48.829732 INFO     log_clustering : Processed 1318 lines in 0.02 seconds (64798.84 lines/second).                                              log_clustering.py:226
Zookeeper_2k.log ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
                                                                               Log Clusters
┏━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Count ┃ Char Size (KB) ┃ Template                                                                                                                                      ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│   313 │             43 │ 2015-07-29 <TIME> - WARN [SendWorker:188978561024:QuorumCnxManager$SendWorker@679] - Interrupted while waiting for message on queue           │
│   289 │             43 │ 2015-07-29 <TIME> - WARN [RecvWorker:188978561024:QuorumCnxManager$RecvWorker@762] - Connection broken for id 188978561024, my id = <*> error │
│       │                │ =                                                                                                                                             │
│   265 │             30 │ 2015-07-29 <TIME> - WARN [RecvWorker:188978561024:QuorumCnxManager$RecvWorker@765] - Interrupting SendWorker                                  │
│   262 │             31 │ 2015-07-29 <TIME> - WARN <*> - Send worker leaving thread                                                                                     │
│    45 │              7 │ <*> <TIME> - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@368] - Cannot open channel to 2 at election address /<IP>:3888    │
│    39 │              7 │ <*> <TIME> - WARN [NIOServerCxn.Factory:<IP>/<IP>:2181:ZooKeeperServer@793] - Connection request from old client <*> will be dropped if       │
│       │                │ server is in r-o mode                                                                                                                         │
│    37 │              4 │ <*> <TIME> - WARN [NIOServerCxn.Factory:<IP>/<IP>:2181:NIOServerCnxn@349] - caught end of stream exception                                    │
│    22 │              3 │ 2015-08-25 <TIME> - WARN <*> - Cannot open channel to 3 at election address /<IP>:3888                                                        │
│    14 │              2 │ 2015-08-24 <TIME> - WARN <*> - Cannot open channel to 3 at election address /<IP>:3888                                                        │
│     3 │              0 │ 2015-07-30 <TIME> - WARN [WorkerSender[myid=1]:QuorumCnxManager@368] - Cannot open channel to <*> at election address /<IP>:3888              │
│     3 │              0 │ 2015-08-20 <TIME> - WARN [NIOServerCxn.Factory:<IP>/<IP>:2181:NIOServerCnxn@354] - Exception causing close of session <HEX> due to            │
│       │                │ java.io.IOException: ZooKeeperServer not running                                                                                              │
│     1 │              0 │ 2015-07-30 <TIME> - WARN [LearnerHandler-/<IP>:35276:LearnerHandler@575] - ******* GOODBYE /<IP>:35276 ********                               │
│     1 │              0 │ 2015-07-30 <TIME> - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@765] - Interrupting SendWorker                                             │
│     1 │              0 │ 2015-08-25 <TIME> - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@762] - Connection broken for id 3, my id = 1, error =                      │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [LearnerHandler-/<IP>:52264:LearnerHandler@575] - ******* GOODBYE /<IP>:52264 ********                               │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [LearnerHandler-/<IP>:52308:LearnerHandler@575] - ******* GOODBYE /<IP>:52308 ********                               │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [LearnerHandler-/<IP>:59060:LearnerHandler@575] - ******* GOODBYE /<IP>:59060 ********                               │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [LearnerHandler-/<IP>:59103:LearnerHandler@575] - ******* GOODBYE /<IP>:59103 ********                               │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [LearnerHandler-/<IP>:57247:LearnerHandler@575] - ******* GOODBYE /<IP>:57247 ********                               │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [LearnerHandler-/<IP>:57319:LearnerHandler@575] - ******* GOODBYE /<IP>:57319 ********                               │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [LearnerHandler-/<IP>:52476:LearnerHandler@575] - ******* GOODBYE /<IP>:52476 ********                               │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [LearnerHandler-/<IP>:59203:LearnerHandler@575] - ******* GOODBYE /<IP>:59203 ********                               │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [LearnerHandler-/<IP>:59211:LearnerHandler@575] - ******* GOODBYE /<IP>:59211 ********                               │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [LearnerHandler-/<IP>:52502:LearnerHandler@575] - ******* GOODBYE /<IP>:52502 ********                               │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [LearnerHandler-/<IP>:57458:LearnerHandler@575] - ******* GOODBYE /<IP>:57458 ********                               │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [LearnerHandler-/<IP>:57502:LearnerHandler@575] - ******* GOODBYE /<IP>:57502 ********                               │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [LearnerHandler-/<IP>:52703:LearnerHandler@575] - ******* GOODBYE /<IP>:52703 ********                               │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [LearnerHandler-/<IP>:59406:LearnerHandler@575] - ******* GOODBYE /<IP>:59406 ********                               │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [LearnerHandler-/<IP>:57580:LearnerHandler@575] - ******* GOODBYE /<IP>:57580 ********                               │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [LearnerHandler-/<IP>:52818:LearnerHandler@575] - ******* GOODBYE /<IP>:52818 ********                               │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [LearnerHandler-/<IP>:52844:LearnerHandler@575] - ******* GOODBYE /<IP>:52844 ********                               │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [LearnerHandler-/<IP>:57653:LearnerHandler@575] - ******* GOODBYE /<IP>:57653 ********                               │
│     1 │              0 │ 2015-07-31 <TIME> - WARN [SendWorker:1:QuorumCnxManager$SendWorker@679] - Interrupted while waiting for message on queue                      │
│     1 │              0 │ 2015-08-07 <TIME> - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@368] - Cannot open channel to 3 at election address        │
│       │                │ /<IP>:3888                                                                                                                                    │
│     1 │              0 │ 2015-08-20 <TIME> - WARN [LearnerHandler-/<IP>:42241:Leader@576] - First is <HEX>                                                             │
│     1 │              0 │ 2015-07-29 <TIME> - WARN [WorkerSender[myid=3]:QuorumCnxManager@368] - Cannot open channel to 2 at election address /<IP>:3888                │
│     1 │              0 │ 2015-07-30 <TIME> - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@762] - Connection broken for id 1, my id = 3, error =                      │
└───────┴────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Launch gool and filter lines on specified time

gool   /home/godardo/git/gool/tests/data/log/Apache_2k.log --time_format '%a %b %d %H:%M:%S %Y' --time_pattern '\[(\w{3} \w{3} \d{2} \d{2}:\d{2}:\d{2} \d{4})' --time_min 'Mon Dec 05 18:46:00 2005' --time_max 'Mon Dec 05 19:00:00 2005'

22:59:35.061049 INFO     log_clustering : Loading configuration from /home/godardo/.drain3.ini                                                                                     log_clustering.py:266
22:59:35.069775 INFO     log_clustering : Running clustering.                                                                                                                      log_clustering.py:575
22:59:35.091847 INFO     logs_miner : Reached a log line with time after the specified maximum time. Stopping processing further lines.                                                logs_miner.py:155
22:59:35.092984 INFO     logs_miner : Processed 9 lines in 0.02 seconds (470.37 lines/second).                                                                                         logs_miner.py:164
Apache_2k.log ━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   9%
                                                  Log Clusters
┏━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Count ┃ Char Size (KB) ┃ Template                                                                             ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│     3 │              0 │ [Mon Dec 05 <TIME>] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties │
│     3 │              0 │ [Mon Dec 05 <TIME>] [error] mod jk child workerEnv in error state 6                  │
│     2 │              0 │ [Mon Dec 05 <TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 8         │
│     1 │              0 │ [Mon Dec 05 <TIME>] [notice] jk2 init() Found child 6740 in scoreboard slot 7        │
└───────┴────────────────┴──────────────────────────────────────────────────────────────────────────────────────┘

Compare logs with a baseline

Comparison in two different files :

gool /home/godardo/git/gool/tests/data/log/Apache_2K_MonDec05.log  --display_common --baseline /home/godardo/git/gool/tests/data/log/Apache_2K_SunDec04.log

23:04:06.371468 INFO     log_clustering : Loading configuration from /home/godardo/git/gool/tests/data/drain3_compare_baseline.ini                                                 log_clustering.py:266
23:04:06.374091 INFO     log_clustering : Running clustering.                                                                                                                      log_clustering.py:575
23:04:06.389677 INFO     logs_miner : Processed 949 lines in 0.01 seconds (80002.33 lines/second).                                                                                     logs_miner.py:164
Apache_2K_MonDec05.log ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
23:04:06.391483 INFO     log_clustering : Running baseline clustering.                                                                                                             log_clustering.py:588
23:04:06.406525 INFO     logs_miner : Processed 1051 lines in 0.01 seconds (79220.32 lines/second).                                                                                    logs_miner.py:164
Apache_2K_SunDec04.log ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
                         Missing /baseline
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Template                                                            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ [<TIME>] [notice] jk2 init() Found child 1568 in scoreboard slot 13 │
│ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 11  │
└─────────────────────────────────────────────────────────────────────┘

                         Added /baseline
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Template                                                           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 13 │
└────────────────────────────────────────────────────────────────────┘

                                Common with baseline
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Template                                                                         ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ [<TIME>] [error] [client <IP>] Directory index forbidden by rule: /var/www/html/ │
│ [<TIME>] [error] jk2 init() Can't find child <*> in scoreboard                   │
│ [<TIME>] [error] mod jk child init 1 -2                                          │
│ [<TIME>] [error] mod jk child workerEnv in error state <*>                       │
│ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 10               │
│ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 12               │
│ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 6                │
│ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 7                │
│ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 8                │
│ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 9                │
│ [<TIME>] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties        │
└──────────────────────────────────────────────────────────────────────────────────┘

                                            Baseline Log Clusters
┏━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Count ┃ Char Size (KB) ┃ Template                                                                         ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│   298 │             27 │ [<TIME>] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties        │
│   281 │             21 │ [<TIME>] [error] mod jk child workerEnv in error state <*>                       │
│    94 │              8 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 6                │
│    93 │              7 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 8                │
│    89 │              7 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 7                │
│    87 │              7 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 9                │
│    59 │              5 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 10               │
│    18 │              1 │ [<TIME>] [error] [client <IP>] Directory index forbidden by rule: /var/www/html/ │
│    16 │              1 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 11               │
│     6 │              0 │ [<TIME>] [error] jk2 init() Can't find child <*> in scoreboard                   │
│     6 │              0 │ [<TIME>] [error] mod jk child init 1 -2                                          │
│     3 │              0 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 12               │
│     1 │              0 │ [<TIME>] [notice] jk2 init() Found child 1568 in scoreboard slot 13              │
└───────┴────────────────┴──────────────────────────────────────────────────────────────────────────────────┘

                                                Log Clusters
┏━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Count ┃ Char Size (KB) ┃ Template                                                                         ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│   271 │             24 │ [<TIME>] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties        │
│   258 │             19 │ [<TIME>] [error] mod jk child workerEnv in error state <*>                       │
│   105 │              8 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 7                │
│   101 │              8 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 8                │
│    95 │              8 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 6                │
│    73 │              6 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 9                │
│    16 │              1 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 10               │
│    14 │              1 │ [<TIME>] [error] [client <IP>] Directory index forbidden by rule: /var/www/html/ │
│     6 │              0 │ [<TIME>] [error] jk2 init() Can't find child <*> in scoreboard                   │
│     6 │              0 │ [<TIME>] [error] mod jk child init 1 -2                                          │
│     2 │              0 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 12               │
│     2 │              0 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 13               │
└───────┴────────────────┴──────────────────────────────────────────────────────────────────────────────────┘

Comparison in the same file:

gool --cfg-file /home/godardo/git/gool/tests/data/drain3_compare_baseline.ini --filter ".*Mon Dec 05.*" --base-filter '.*Sun Dec 04.*' '/home/godardo/git/gool/tests/data/log/Apache_2k.log'  --baseline  '/home/godardo/git/gool/tests/data/log/Apache_2k.log'

gool  /home/godardo/git/gool/tests/data/log/Apache_2k.log --cfg-file /home/godardo/git/gool/tests/data/drain3_compare_baseline.ini  --time-format  "%a %b %d %H:%M:%S %Y" --time-pattern "(\\w{3} \\w{3} \\d{2} \\d{2}:\\d{2}:\\d{2} \\d{4})" --time-min "Mon Dec 05 00:00:00 2005" --time-max "Tue Dec 06 00:00:00 2005" --base-time-min "Sun Dec 04 00:00:00 2005" --base-time-max "Mon Dec 05 00:00:00 2005"  --baseline /home/godardo/git/gool/tests/data/log/Apache_2k.log

13:21:05.732175 INFO     log_clustering : Running clustering.                                                                                                                                                                         log_clustering.py:346
13:21:05.734431 INFO     logs_miner : Loading configuration from /home/godardo/git/gool/tests/data/drain3_compare_baseline.ini                                                                                                             logs_miner.py:29
13:21:05.754302 INFO     logs_miner : Processed 949 lines in 0.02 seconds (60920.41 lines/second).                                                                                                                                        logs_miner.py:213
Apache_2k.log ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
13:21:05.755925 INFO     log_clustering : Running baseline clustering.                                                                                                                                                                log_clustering.py:364
13:21:05.756522 INFO     logs_miner : Loading configuration from /home/godardo/git/gool/tests/data/drain3_compare_baseline.ini                                                                                                             logs_miner.py:29
13:21:05.776119 INFO     logs_miner : Processed 1051 lines in 0.02 seconds (59559.73 lines/second).                                                                                                                                       logs_miner.py:213
Apache_2k.log ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
                         Missing /baseline
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Template                                                            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ [<TIME>] [notice] jk2 init() Found child 1568 in scoreboard slot 13 │
│ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 11  │
└─────────────────────────────────────────────────────────────────────┘

                         Added /baseline
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Template                                                           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 13 │
└────────────────────────────────────────────────────────────────────┘

                                                Log Clusters
┏━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Count ┃ Char Size (KB) ┃ Template                                                                         ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│   271 │             24 │ [<TIME>] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties        │
│   258 │             19 │ [<TIME>] [error] mod jk child workerEnv in error state <*>                       │
│   105 │              8 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 7                │
│   101 │              8 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 8                │
│    95 │              8 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 6                │
│    73 │              6 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 9                │
│    16 │              1 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 10               │
│    14 │              1 │ [<TIME>] [error] [client <IP>] Directory index forbidden by rule: /var/www/html/ │
│     6 │              0 │ [<TIME>] [error] jk2 init() Can't find child <*> in scoreboard                   │
│     6 │              0 │ [<TIME>] [error] mod jk child init 1 -2                                          │
│     2 │              0 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 12               │
│     2 │              0 │ [<TIME>] [notice] jk2 init() Found child <*> in scoreboard slot 13               │
└───────┴────────────────┴──────────────────────────────────────────────────────────────────────────────────┘

More option details

gool --help

Merge several log files ordered by time

gool-merge takes one or more log files and prints their lines to stdout, interleaved and ordered by timestamp. It is handy to reconstruct a single timeline out of logs coming from several processes, services or hosts.

Each input file is expected to already be internally sorted by timestamp (a normal log file); gool-merge does a streaming merge of the files, it does not re-sort lines within a single file.
Lines on which no timestamp can be found or parsed are skipped (ignored) rather than failing the merge; how many were skipped per file is logged once at the end.
When two lines share the exact same timestamp, the line coming from the file listed first on the command line is output first.
filter, time_min, time_max, time_pattern and time_format behave exactly like their gool clustering counterparts (see Configuration above) and are used both to select which lines to keep and to extract the timestamp used for ordering.
If none of time_pattern, time_format, time_min or time_max is provided, each line is tried against "%H:%M:%S.%f", "%H:%M:%S,%f" then "%H:%M:%S" in turn (same as gool clustering's documented default). Since none of these carry a date, files spanning more than one day will only be ordered by time-of-day unless you pass an explicit --time-pattern/--time-format that also captures the date (see example below).
add-filename (-n): if set, appends the source filename in brackets, e.g. [app.log], at the end of each output line.

gool-merge tests/data/log/Crafted_logs.log tests/data/log/Crafted_logs_2.log --add-filename

12:00:12.121 | INFO  | first info 1234 [Crafted_logs.log]
12:00:12.121 | INFO  | from second file A [Crafted_logs_2.log]
12:00:12.221 | INFO  | first info [Crafted_logs.log]
12:00:14.121 | INFO  | first info 8931 [Crafted_logs.log]
12:00:14.121 | INFO  | second info [Crafted_logs.log]
12:00:14.500 | ERROR | from second file B [Crafted_logs_2.log]
13:00:10.000 | INFO  | from second file C [Crafted_logs_2.log]
13:00:14.121 | ERROR | second info [Crafted_logs.log]

Note the two files' 12:00:12.121 lines are tied: Crafted_logs.log was listed first on the command line, so its line comes first. The Crafted_logs_2.log line reading "no timestamp here at all" has no timestamp and was silently skipped (reported once on stderr as "Skipped 1 line(s) without a matching timestamp").

Only keep errors:

gool-merge tests/data/log/Crafted_logs.log tests/data/log/Crafted_logs_2.log --filter '.*ERROR.*'

12:00:14.500 | ERROR | from second file B
13:00:14.121 | ERROR | second info

Files spanning several days need a timestamp pattern/format that also captures the date, exactly like gool:

gool-merge tests/data/log/Apache_2k_MonDec05.log tests/data/log/Apache_2k_SunDec04.log \
  --time-format '%a %b %d %H:%M:%S %Y' --time-pattern '\[(\w{3} \w{3} \d{2} \d{2}:\d{2}:\d{2} \d{4}).*'

[Sun Dec 04 04:47:44 2005] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties
[Sun Dec 04 04:47:44 2005] [error] mod_jk child workerEnv in error state 6
[Sun Dec 04 04:51:08 2005] [notice] jk2_init() Found child 6725 in scoreboard slot 10
...
[Mon Dec 05 01:04:31 2005] [error] [client 218.62.18.218] Directory index forbidden by rule: /var/www/html/
...

More gool-merge option details

gool-merge --help

Example of a drain3.ini file

gool try to load .drain3 file from your home. Otherwise you can use --cfg-file option. Without any configuration, gool will try to mask HEX, IPs and times.

Below an example of configuration file :

[MASKING]
masking = [
    {"regex_pattern":"((?<=[^A-Za-z0-9])|^)(0[xX][0-9a-fA-F]+)((?=[^A-Za-z0-9])|$)", "mask_with": "HEX"},
    {"regex_pattern":"(\\d{2}:\\d{2}:\\d{2}(.\\d+|))", "mask_with": "TIME"},
    {"regex_pattern":"((?<=[^A-Za-z0-9])|^)(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})((?=[^A-Za-z0-9])|$)", "mask_with": "IP"}
    ]
mask_prefix = <:
mask_suffix = :>

[DRAIN]
sim_th = 0.9
depth = 7
max_children = 200
extra_delimiters = ["_"]

Source Setup

The gool repository uses uv and git. Version is taken from git tag. The repo provide setup for VSCode.

Below the more useful commands.

Setup the uv virtual environment and install pre-commit hooks :

make install

Non regression tests on all supported environments :

tox -p

Generate and launch the documentation server :

make docs

All available commands :

make help

Repository initiated with fpgmaas/cookiecutter-uv.

Metadata

Release files for gool 1.3.7

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for gool 1.3.7
File	Size	Uploaded
gool-1.3.7.tar.gz	170.4 kB	Oct 10, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for gool 1.3.7
File	Interpreter	ABI	Platform	Reset
gool-1.3.7-py3-none-any.whl 22.9 kB Oct 10, 2026	Python 3	none	any	Details

Total release size: 193.3 kB

Release files / gool-1.3.7.tar.gz

Download URL	gool-1.3.7.tar.gz
Size	170.4 kB
Tags	Source
SHA-256 checksum How to use checksums	`4dea67012305e12003582cb8a24099a2a84a002cfcee35a21177aca0a0b6d08a`
BLAKE2b-256 checksum How to use checksums	`96d44b678c483793e021eddd270f52bf8557cf761a4eb25de6c5c90ac02da2da`
Upload date	Oct 10, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.14.4`

Release files / gool-1.3.7-py3-none-any.whl

Download URL	gool-1.3.7-py3-none-any.whl
Size	22.9 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`ce64837c93caccfeb2b5596fbeb8d96b9e864d4e9cbd8d87ced02ff4abf2534d`
BLAKE2b-256 checksum How to use checksums	`53965bb2a5d17d6a13db8185dce662f4ba6351c9f2ea8feb491de4415ccbcc89`
Upload date	Oct 10, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.14.4`