A simple Django project profiler for timing HTTP requests.
This package now requires Python 3.9 and Django 3.2 and above.
A very simple request profiler for Django.
Premature optimization is the root of all evil.
There are a lot of very good, and complete, python and django profilers available. They can give you detailed stack traces and function call timings, output all the SQL statements that have been run, the templates that have been rendered, and the state of any / all variables along the way. These tools are great for optimisation of your application, once you have decided that the time is right.
django-request-profiler is not intended to help you optimise, but to help you decide whether you need to optimise in the first place. It is complimentary.
Small enough to run in production
Able to configure profiling at runtime
Configurable to target specific URLs or users
Record basic request metadata:
Request path, remote addr, user-agent
Response status code, content length
Django user and session keys (if appropriate)
Database query count (if DEBUG=True)
It doesn’t need to record all the inner timing information - the goal is to have a system that can be used to monitor site response times, and to identify problem areas ahead of time.
The profiler itself runs as Django middleware, and it simply starts a timer when it first sees the request, and stops the timer when it is finished with the response. It should be installed as the first middleware in MIDDLEWARE_CLASSES in order to record the maximum duration.
It hooks into the process_request method to start the timer, the process_view method to record the view function name, and the process_response method to stop the timer, record all the request information and store the instance.
The profiler is controlled by adding RuleSet instances which are used to filter which requests are profiled. There can be many, overlapping, RuleSets, but if any match, the request is profiled. The RuleSet model defines two core matching methods:
1. uri_regex - in order to profile a subset of the site, you can supply a regex which is used match the incoming request path. If the url matches, the request can be profiled.
2. user_filter_type - there are three choices here - profile all users, profile only authenticated users, and profile authenticated users belonging to a given Group - e.g. create a groups called “profiling” and add anyone you want to profile.
These filter properties are an AND (must pass the uri and user filter), but the rules as a group are an OR - so if a request passes all the filters in any rule, then it’s profiled.
These filters are pretty blunt, and there are plenty of use cases where you may want more sophisticated control over the profiling. There are two ways to do this. The first is a setting, REQUEST_PROFILER_GLOBAL_EXCLUDE_FUNC, which is a function that takes a request as the single argument, and must return True or False. If it returns False, the profile is cancelled, irrespective of any rules. The primary use case for this is to exclude common requests that you are not interested in, e.g. from search engine bots, or from Admin users etc. The default for this function is to prevent admin user requests from being profiled.
The second control is via the cancel() method on the ProfilingRecord, which is accessible via the request_profile_complete signal. By hooking in to this signal you can add additional processing, and optionally cancel the profiler. A typical use case for this is to log requests that have exceeded a set request duration threshold. In a high volume environment you may want to, for instance, only profile a random subset of all requests.
from django.dispatch import receiver from request_profiler.signals import request_profile_complete @receiver(request_profiler_complete) def on_request_profile_complete(sender, **kwargs): profiler = kwargs.get('instance') if profiler.elapsed > 2: # log long-running requests # NB please don't use 'print' for real - use logging print u"Long-running request warning: %s" % profiler else: # calling cancel means that it won't be saved to the db profiler.cancel()
An additional scenario where you may want to use the signal is to store the profiler records async - say if you are recording every request for a short period, and you don’t want to add unnecessary inline database write operations. In this case you can use the stop() method, which will prevent the middleware from saving it directly (it will only save records where profiler.is_running is true, and both cancel and stop set it to false).
from django.dispatch import receiver from request_profiler.signals import request_profile_complete @receiver(request_profiler_complete) def on_request_profile_complete(sender, **kwargs): profiler = kwargs.get('instance') # stop the profiler to prevent it from being saved automatically profiler.stop() assert not profiler.is_running # add a job to a queue to perform the save itself queue.enqueue(profiler.save)
For use as the app in Django project, use pip:
$ pip install django-request-profiler # For hacking on the project, pull from Git: $ git pull firstname.lastname@example.org:yunojuno/django-request-profiler.git
The app installer contains a test suite that can be run using the Django test runner:
$ pip install -r requirements.txt $ python manage.py test test_app request_profiler
If you want to test coverage you’ll need to add some dependencies:
$ pip install coverage django-coverage $ python manage.py test_coverage test_app request_profiler
The tests also run using tox:
$ pip install tox $ tox
Note: To test with a custom user model, you should override the default User model by providing a value for the AUTH_USER_MODEL (in testapp/settings) setting that references a custom model
The tests run on Travis on commits to master.
Once installed, add the app and middleware to your project’s settings file. In order to add the database tables, you should run the migrate command:
$ python manage.py migrate request_profiler
NB the middleware must be the first item in MIDDLEWARE_CLASSES.
INSTALLED_APPS = ( 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'request_profiler', ) MIDDLEWARE_CLASSES = [ # this package's middleware 'request_profiler.middleware.ProfilingMiddleware', # default django middleware 'django.middleware.common.CommonMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', ]
To configure the app, open the admin site, and add a new request profiler ‘Rule set’. The default options will result in all non-admin requests being profiled.
MIT (see LICENCE)
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Hashes for django_request_profiler-1.1.tar.gz
Hashes for django_request_profiler-1.1-py3-none-any.whl