k3logcollector
Scan log files on local machine, collector all interested logs, and send to somewhere for display.
k3logcollector is a component of pykit3 project: a python3 toolkit set.
Name
logcollector
Scan log files on local machine, collector all interested logs, and send to somewhere for display.
Description
We may want to see all error logs on all machines, so we need to collector logs, and save it in somewhere. This module is used to collector logs on one machine.
Normally, same error info will be loged repeatedly, we do not want to save duplicated log info, so logs produced by same source file at same line number in one second will be combined.
Each flush, once per second, sends only the logs of the newest 5 timestamps
and drops older ones, unless the log's conf sets merge to False. So a
backlog, such as the 2 MB that the scanner reads back when it starts, does not
keep the sender behind.
Conf
configuration for log files. It is a dict, the key is the log name, the value is the configuration for the log.
file_path
the path of the log file.
is_first_line
is a callback function.
The argument to this function is a line in log file, if the line is the
first line of a log, then return True, otherwise return False.
get_level
is a callback function. The argument to this function is the complete log string, which may contains multiple lines. It should return the level of the log, which is a string.
parse
is a callback function. The argument to this function is the complete log string, which may contains multiple lines. It should return a dict contains following fields.
-
log_ts: the timestamp of this log, such as
1523936052, is a integer. -
level: the level of this log, such as 'info'.
-
source_file: the source file in which the log was produced.
-
line_number: the number of the line at which the log was produced.
level
is a list, used to specify the interested log levels.
Install
pip install k3logcollector
Synopsis
import k3logcollector
def send_log(log_entry):
print("send the log entry to database or other place.")
def is_first_line(line):
print("return True if the line is the first line of a log,")
print("otherwise return False.")
def get_level(log_str):
print("return the level of the log.")
def parse(log_str):
print("parse the log.")
conf = {
"front_error_log": {
"file_path": "path/to/log/file/xxx.error.log",
"level": ["warn", "error"],
"is_first_line": is_first_line,
"get_level": get_level,
"parse": parse,
},
}
kwargs = {
"node_id": "123abc",
"node_ip": "1.2.3.4",
"send_log": send_log,
"conf": conf,
}
k3logcollector.run(**kwargs)
Author
Zhang Yanpo (张炎泼) drdr.xp@gmail.com
Copyright and License
The MIT License (MIT)
Copyright (c) 2015 Zhang Yanpo (张炎泼) drdr.xp@gmail.com
Metadata
Release files for k3logcollector 0.1.8
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| k3logcollector-0.1.8.tar.gz | 9.3 kB | Without provenance | Details for k3logcollector-0.1.8.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| k3logcollector-0.1.8-py3-none-any.whl | Python 3 | none | any | Details for k3logcollector-0.1.8-py3-none-any.whl |
Total release size: 18.6 kB
Release files / k3logcollector-0.1.8.tar.gz
| Download URL | k3logcollector-0.1.8.tar.gz |
|---|---|
| Size | 9.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1c3a4d2d1111f9f68f4978562befd937e40d9d4adb2c949c4a30dab97cd64179
|
|
BLAKE2b-256 checksum How to use checksums |
fd3ef3754e47d8cfb57a666f1a9ce78844bc8a9f95018c03ebb91b157550df42
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.8
|
Release files / k3logcollector-0.1.8-py3-none-any.whl
| Download URL | k3logcollector-0.1.8-py3-none-any.whl |
|---|---|
| Size | 9.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
58c77f1494e783877d065c0a5c27c8d952c0d9778529183376f9b706402a96e0
|
|
BLAKE2b-256 checksum How to use checksums |
ff0bc5e95c7d41a79a0b725c29c9afc481915c44680f9f7704363dc3646ac6ea
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.8
|
Security for k3logcollector 0.1.8
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.