HTTP log monitoring
This project allows watching and logging all HTTP traffic of a system, generating log in w3c log format, showing statistics about total requests, maximum hits, requests in a timespan and alerting when traffic is above a customizable threshold.
- Python and preferably Linux.
You can execute
simulate.shto run a simulation of how this project works. The simulation setup is customisable, feel free to play with it.
Make sure to install the package by runing :
python setup.py install
pip install monilog
- To run the monitoring in your own log file, run:
monitoring --file /path/to/your/file --threshold 10
- To customize the log generation, run:
log_generator --rates 9 11 8 --durations 150 150 150
rates being the number of requests per second for each step of the simulation and
durations being the durations of the corresponding simulation steps.
- To execute the tests , run :
nosetests --with-coverage --cover-package=monilog
Attention: The monitoring is stopped when no new logs are written to the log
MAX_IDLE_TIME set by default to 2 minutes. This is added to manage stopping
the monitoring automatically, particularly when doing limited time simulations.
This is a first working solution for http log monitoring. Many improvements can be added :
- Managing threaded access to the log file using cross-platform file locking. The current implementation is tested on Linux and may cause errors in Windows.
- Enhancing the display of the log analysis and statistics. For now, the monitoring results are written
to standard output and to a log file with the naming convention
simulation-<timestamp>.log. A better setup would be to customize the GUI using npyscreen for instance.
- It is also possible to build a live dashboard consuming
simulation-<timestamp>.logdata and mapping it to graphs.
- Pushing alerting notifications by email or SMS to admins / owners of the monitored system.
- Adding more relevant statistics to the analysis of the website and handle timezone changes.
- Writing extensive unit and integration tests.
- For a higher scale of data requiring high availability in a production setup, a more robust solution would be indexing the logs in ElasticSearch and building Kibana dashboards, with a stream-processing platform such as Kafka.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size monilog-0.1.4-py3-none-any.whl (9.2 kB)||File type Wheel||Python version py3||Upload date||Hashes View|
|Filename, size monilog-0.1.4.tar.gz (6.2 kB)||File type Source||Python version None||Upload date||Hashes View|