nb_log_file_handler
multi process safe log file handler,both time and size rotate,benchmark fast than concurrent_log_handler 100 times
nb_log_file_handler 是多进程安全切割,同时按时间和大小切割的FileHandler,性能远超 concurrent_log_handler.ConcurrentRotatingFileHandler
安装
pip install nb_log_file_handler
nb_log_file_handler 实现原理,
nb_log_file_handler 在win上采用每隔0.1秒批量写入文件,atexit钩子对程序即将结束后的剩余待写入的消息写到文件中。 linux的文件io性能本身比较好,加上fork 子进程不支持 atexit 触发执行,所以linux上使用单个消息就写入。
nb_log_file_handler 性能远超 concurrent_log_handler.ConcurrentRotatingFileHandler
1、nb_log_file_handler使用方式:
代码如下,和filehandler用法相同,导入 NbLogFileHandler
import multiprocessing
import logging
import time
from nb_log_file_handler import NbLogFileHandler
logger = logging.getLogger('hello')
fh = NbLogFileHandler(file_name='xx3.log',log_path='./',max_bytes=1000*1000,back_count=3)
logger.addHandler(fh)
# logger.addHandler(logging.StreamHandler())
logger.setLevel(logging.DEBUG)
def f():
for i in range(10000):
logger.info(f'{i}aaaaa'*20)
2、各种按文件/时间大小切割的fileHander对比,
为了测试多进程按文件大小切割安全的复现,所以所有maxBytes按照1000*1000字节,即1M进行切割。
2.1、对比logging内置的 logging.handlers.RotatingFileHandler
logging.handlers.RotatingFileHandler 多进程按大小切割完全不可行,切割时候疯狂报错
import multiprocessing
import logging.handlers
import time
logger = logging.getLogger('hello')
fh = logging.handlers.RotatingFileHandler('xx4.log',maxBytes=1000*1000,backupCount=3)
logger.addHandler(fh)
# logger.addHandler(logging.StreamHandler())
logger.setLevel(logging.DEBUG)
def f():
for i in range(10000):
logger.info(f'{i}aaaaa'*20)
if __name__ == '__main__':
t1 = time.time()
ps = []
for j in range(10):
p = multiprocessing.Process(target=f)
ps.append(p)
p.start()
for p in ps:
p.join()
print(time.time()-t1)
这个代码使用 文件handler选择原生自带的 logging.handlers.RotatingFileHandler 会疯狂报错,因为进程a在达到大小切割改名日志文件时候,进程b并不知情,报错如下:
PermissionError: [WinError 32] 另一个程序正在使用此文件,进程无法访问。: 'D:\\codes\\nb_log_file_handler\\tests_nb_log_file_handler\\xx4.log' -> 'D:\\codes\\nb_log_file_handler\\tests_nb_log_file_handler\\xx4.log.1'
所以一般多进程写入同一个日志文件,并支持切割,那么久不能使用logging自带的RotatingFileHandler,要使用第三方包的filehandler。
2.2、对比小有名气的多进程切割安全的三方包 concurrent_log_handler
concurrent_log_handler.ConcurrentRotatingFileHandler
import multiprocessing
import logging
import time
from concurrent_log_handler import ConcurrentRotatingFileHandler
logger = logging.getLogger('hello')
fh = ConcurrentRotatingFileHandler('xx2.log',maxBytes=1000*1000,backupCount=3)
logger.addHandler(fh)
# logger.addHandler(logging.StreamHandler())
logger.setLevel(logging.DEBUG)
def f():
for i in range(10000):
logger.info(f'{i}aaaaa'*20)
if __name__ == '__main__':
t1 = time.time()
ps = []
for j in range(10):
p = multiprocessing.Process(target=f)
ps.append(p)
p.start()
for p in ps:
p.join()
print(time.time()-t1)
concurrent_log_handler这个包在windows上性能无法忍受,10进程写入10000次需要263秒,性能惨不忍睹。这个包在linux上性能还可以接受。
2.3、 nb_log_file_handler.NbLogFileHandler 按时间和大小多进程安全切割,性能远远的暴击 concurrent_log_handler
import multiprocessing
import logging
import time
from nb_log_file_handler import NbLogFileHandler
logger = logging.getLogger('hello')
fh = NbLogFileHandler(file_name='xx3.log',log_path='./',max_bytes=1000*1000,back_count=3)
logger.addHandler(fh)
# logger.addHandler(logging.StreamHandler())
logger.setLevel(logging.DEBUG)
def f():
for i in range(10000):
logger.info(f'{i}aaaaa'*20)
if __name__ == '__main__':
t1 = time.time()
ps = []
for j in range(10):
p = multiprocessing.Process(target=f)
ps.append(p)
p.start()
for p in ps:
p.join()
print(time.time()-t1)
nb_log_file_handler.NbLogFileHandler 10进程写入10000次只需要1.3秒,nb_log_file_handler 性能远远的暴击三方包 concurrent_log_handler
Metadata
Release files for nb-log-file-handler 0.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| nb_log_file_handler-0.2.tar.gz | 5.2 kB | Without provenance | Details for nb_log_file_handler-0.2.tar.gz |
Release files / nb_log_file_handler-0.2.tar.gz
| Download URL | nb_log_file_handler-0.2.tar.gz |
|---|---|
| Size | 5.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9b6a0a2a06b821e8500e485ac34d2ed53171108ef0eb8a43a4c7f86c03591cd8
|
|
BLAKE2b-256 checksum How to use checksums |
2c93ee7ab21e97415d663809c5e1ef7811516e6ab3cf3728c77ecf219f420d0f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.12
|
Security for nb-log-file-handler 0.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.