Skip to main content

Library of web access log analysis

Project description

alternate text

Travis CI build status https://coveralls.io/repos/github/Edinburgh-Genome-Foundry/lala/badge.svg?branch=master

Lala is a Python library for access log analysis. It provides a set of methods to retrieve, parse and analyze access logs (only from NGINX for now), and makes it easy to plot geo-localization or time-series data. Think of it as a simpler, Python-automatable version of Google Analytics, to make reports like this:

alternate text

Usage

from lala import WebLogs
weblogs, errored_lines = WebLogs.from_nginx_weblogs('access_logs.txt')

Similarly, to fetch logs on a distant server (for which you have access keys) you would write:

from lala import get_remote_file_content, WebLogs

logs= lala.get_remote_file_content(
    host="cuba.genomefoundry.org", user='root',
    filename='/var/log/nginx_cuba/access.log'
)
weblogs, errors = WebLogs.from_nginx_weblogs(logs.split('\n'))

Now weblogs is a scpecial kind of Pandas dataframe where each row is one server access, with fields such as IP, date, referrer, country_name, etc.

alternate text

The web logs can therefore be analyzed using any of Pandas’ built-in filtering and plotting functions. The WebLogs class also provides additional methods which are particularly useful to analyse web logs, for instance to plot pie-charts:

ax, country_values = weblogs.plot_piechart('country_name')
alternate text

Next we plot the location (cities) providing the most connexions:

ax = weblogs.plot_geo_positions()
alternate text

We can also restrict the entries to the UK, and plot a timeline of connexions:

uk_entries = weblogs[weblogs.country_name == 'United Kingdom']
ax = uk_entries.plot_timeline(bins_per_day=2)
alternate text

Here is how to get the visitors a list of visitors and visits, sort out the most frequent visitors, find their locations, and plot it all:

visitors = weblogs.visitors_and_visits()
visitors_locations = weblogs.visitors_locations()
frequent_visitors = weblogs.most_frequent_visitors(n_visitors=5)
ax = weblogs.plot_most_frequent_visitors(n_visitors=5)
alternate text

Lala can do more, such as identifying the domain name of the visitors, which can be used to filter out the robots of search engines:

weblogs.identify_ips_domains()
filtered_entries = weblogs.filter_by_text_search(
    terms=['googlebot', 'spider.yandex', 'baidu', 'msnbot'],
    not_in='domain'
)

Lala also plays nicely with the PDF Reports library to let you define report templates such as this one (written in Pug), and then generate this PDF report with the following code:

weblogs.write_report(template_path="path/to/template.pug",
                     target="report_example.pdf")

Installation

You can install lala through PIP

sudo pip install python-lala

Alternatively, you can unzip the sources in a folder and type

sudo python setup.py install

For plotting maps you will need Cartopy which is not always easy to install - it may depend on your system. If you are on Ubuntu 16+, first install the dependencies with:

sudo apt-get install libproj-dev proj-bin proj-data libgeos-dev
sudo pip install cython

License = MIT

lala is an open-source software originally written at the Edinburgh Genome Foundry by Zulko and released on Github under the MIT licence (Copyright 2018 Edinburgh Genome Foundry).

Everyone is welcome to contribute!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

python-lala-0.1.4.tar.gz (25.8 kB view details)

Uploaded Source

Built Distribution

python_lala-0.1.4-py3-none-any.whl (9.8 kB view details)

Uploaded Python 3

File details

Details for the file python-lala-0.1.4.tar.gz.

File metadata

  • Download URL: python-lala-0.1.4.tar.gz
  • Upload date:
  • Size: 25.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/50.0.3 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.6.10

File hashes

Hashes for python-lala-0.1.4.tar.gz
Algorithm Hash digest
SHA256 322eb7ae1fdd9721de5632bf020a0623badedf39502747609b71eb855ea5e09e
MD5 7bd30e3d47667d6a7a232c738ea17121
BLAKE2b-256 5f76c6a3da0e9d308a4ddfc5aa37fbb59f4c20aed68eb9c65c89369ede6ef5b2

See more details on using hashes here.

File details

Details for the file python_lala-0.1.4-py3-none-any.whl.

File metadata

  • Download URL: python_lala-0.1.4-py3-none-any.whl
  • Upload date:
  • Size: 9.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/50.0.3 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.6.10

File hashes

Hashes for python_lala-0.1.4-py3-none-any.whl
Algorithm Hash digest
SHA256 c1d6f0ac7677b8d6dd343d483cf805c476cbc5ff59edc443ba3870375207d090
MD5 a7ed1a77e5b0b2583c410c1daa2f8567
BLAKE2b-256 95fa6839b0828fffb8256af9f64d0d1bcc842c55a208f2363949f0417bf3d41f

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page