Skip to main content

HTTP log monitoring

Project description

Monilog

Build Status codecov PyPI Version

This project allows watching and logging all HTTP traffic of a system, generating log in w3c log format, showing statistics about total requests, maximum hits, requests in a timespan and alerting when traffic is above a customizable threshold.

Requirements

  • Python and preferably Linux.

Usage

  • You can execute simulate.sh to run a simulation of how this project works. The simulation setup is customisable, feel free to play with it.

  • Make sure to install the package by runing :

python setup.py install 

or

pip install monilog
  • To run the monitoring in your own log file, run:
monitoring --file /path/to/your/file --threshold 10
  • To customize the log generation, run:
log_generator --rates 9 11 8 --durations 150 150 150

With rates being the number of requests per second for each step of the simulation and durations being the durations of the corresponding simulation steps.

  • To execute the tests , run :
nosetests --with-coverage --cover-package=monilog

Attention: The monitoring is stopped when no new logs are written to the log file during MAX_IDLE_TIME set by default to 2 minutes. This is added to manage stopping the monitoring automatically, particularly when doing limited time simulations.

Future Improvements

This is a first working solution for http log monitoring. Many improvements can be added :

  • Managing threaded access to the log file using cross-platform file locking. The current implementation is tested on Linux and may cause errors in Windows.
  • Enhancing the display of the log analysis and statistics. For now, the monitoring results are written to standard output and to a log file with the naming convention simulation-<timestamp>.log. A better setup would be to customize the GUI using npyscreen for instance.
  • It is also possible to build a live dashboard consuming simulation-<timestamp>.log data and mapping it to graphs.
  • Pushing alerting notifications by email or SMS to admins / owners of the monitored system.
  • Adding more relevant statistics to the analysis of the website and handle timezone changes.
  • Writing extensive unit and integration tests.
  • For a higher scale of data requiring high availability in a production setup, a more robust solution would be indexing the logs in ElasticSearch and building Kibana dashboards, with a stream-processing platform such as Kafka.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

monilog-0.1.4.tar.gz (6.2 kB view hashes)

Uploaded Source

Built Distribution

monilog-0.1.4-py3-none-any.whl (9.2 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page