django-cachalot 0.5.0

Caches your Django ORM queries and automatically invalidates them.

Caches your Django ORM queries and automatically invalidates them.

Project status:

Currently in beta, do not use in production

Quick start

Requirements

• Django 1.6 or 1.7

• Python 2.6, 2.7, 3.2, 3.3, or 3.4

• locmem or django-redis (memcached coming soon)

• SQLite, PostgreSQL or MySQL (it should work with Oracle, but I don’t have 17.5k$ to test)

Usage

1. pip install django-cachalot

2. Add 'cachalot', to your INSTALLED_APPS

3. Enjoy!

Settings

Setting	Default value	Description
CACHALOT_ENABLED	True	If set to False, disables SQL caching but keeps invalidating to avoid stale cache
CACHALOT_CACHE	'default'	Name of the cache from CACHES used by django-cachalot

These settings can be changed whenever you want. You have to use cachalot_settings as a context manager, a decorator, or simply by changing its attributes:

from cachalot.settings import cachalot_settings

with cachalot_settings(CACHALOT_ENABLED=False):
    # SQL queries are not cached in this block

@cachalot_settings(CACHALOT_CACHE='another_cache')
def your_function():
    # What’s in this function uses another cache

# Globally disables SQL caching until you set it back to True
cachalot_settings.CACHALOT_ENABLED = False

In tests, you can use Django’s testing tools as well as cachalot_settings. They will both work the same.

Limits

Django-cachalot doesn’t cache queries it can’t reliably invalidate. If a SQL query or a part of it is written in pure SQL, it won’t be cached.

That’s why QuerySet.extra with select or where arguments, Model.objects.raw(…), & cursor.execute(…) queries are not cached.

Bug reports, questions, discussion, new features

• If you spotted a bug, please file a precise bug report on GitHub

• If you have a question on how django-cachalot works or to simply discuss, go to our Google group.

• If you want to add a feature:

  • if you have an idea on how to implement it, you can fork the project and send a pull request, but please open an issue first, because someone else could already be working on it

  • if you’re sure that it’s a must-have feature, open an issue

  • if it’s just a vague idea, please ask on google groups before

How django-cachalot works

(If you don’t care/understand, just pretend it’s magic)

Reverse engineering

It’s a lot of Django reverse engineering combined with a strong test suite. Such a test suite is crucial for a reverse engineering project. If some important part of Django changes and breaks the expected behaviour, you can be sure that the test suite will fail.

Monkey patching

django-cachalot modifies Django in place during execution to add a caching tool just before SQL queries are executed. We detect which cache keys must be removed when some data is created/changed/deleted on the database.

What still needs to be done

For version 1.0

• Find out if it’s thread-safe and test it

• Write tests for multi-table inheritance

• Add memcached support

• Handle multiple databases

• Add invalidation on migrations in Django 1.7 (& South?)

In a more distant future

• Add a setting to choose if we cache QuerySet.order_by('?')

• Cache QuerySet.extra if none of set(connection.introspection.table_names()) - set(connection.introspection.django_table_names()) is found in the extra select and where queries

• Add a setting to disable caching on QuerySet.extra when it has select or where rules because we can’t reliably detect other databases (and meta databases like information_schema) on every database backend

• Maybe parse QuerySet.extra with select or where arguments in order to find which tables are implied, and therefore be able to cache them

Legacy

This work is highly inspired of johnny-cache, another easy-to-use ORM caching tool! It’s working with Django <= 1.5. I used it in production during 3 years, it’s an excellent module!

Unfortunately, we failed to make it migrate to Django 1.6 (I was involved). It was mostly because of the transaction system that was entirely refactored.

I also noticed a few advanced invalidation issues when using QuerySet.extra and some complex cases implying multi-table inheritance and related ManyToManyField.