Log actions to Elastic Search via bulk upload
Project description
bes is a flexible bulk uploader for Elastic Search. At the moment, it only supports index uploads over UDP. You can retrieve the logged data from Elastic Search and analyze it dynamically using Kibana (source). This package just makes the initial upload to Elastic Search as painless as possible.
The connection parameters and other configuration options are stored in bes.DEFAULT. Override them as you see fit (e.g. to connect to a remote Elastic Search server):
>>> import bes >>> bes.DEFAULT['host'] = 'my-es-server.example.com'
Then log away:
>>> bes.log(type='record', user='jdoe', action='swims')
If you are so inclined, you can override the DEFAULT index on a per-call basis:
>>> bes.log(index='my-index', type='my-type', user='jdoe', action='bikes')
The log generated by the above will look like:
{ "@timestamp": "2013-09-26T16:34:09.179048", "@version": 1, "action": "bikes", "user": "jdoe" }
following Jordan Sissel’s new format for Logstash:
You should specify a unique type for each record you create, because Elastic Search doesn’t recalculate its mapping if you post a new event that uses an old key with a new data type. If you are only logging a single record type, you can configure a global default:
>>> bes.DEFAULT['type'] = 'record' >>> bes.log(user='jdoe', action='runs')
Django
Although the core of the library is framework-agnostic, we’ll be using this to log events in a Django website. There are a few helpers for extracting information from HttpRequest to make this as easy as possible. Drop it into your views with somthing like:
import django.http imort django.shortcuts import bes.django from polls.models import Poll def detail(request, poll_id): try: p = Poll.objects.get(pk=poll_id) except Poll.DoesNotExist: bes.django.log_user_request_path( request=request, type='404', poll_id=poll_id) raise django.http.Http404 bes.django.log_user_request_path( request=request, type='poll-detail', poll_id=poll_id, poll_name=p.name) return django.shortcutsrender_to_response( 'polls/detail.html', {'poll': p})
You can also override bes’ defaults in your Django config:
BULK_ELASTIC_SEARCH_LOGGING_HOST = 'my-es-server.example.com'
The Django config names match the uppercased DEFAULT keys with a BULK_ELASTIC_SEARCH_LOGGING_ prefix (for example, host → BULK_ELASTIC_SEARCH_LOGGING_HOST).
Testing
Testing uses unittest’s automatic test discovery. Run the test suite with:
$ python -m unittest discover
For Python <3.3, the test suite requires the external mock package, which is bundled as unittest.mock in Python 3.3.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.