Simple and generic model related data prefetch framework for Django solving the "1+N queries" problem that happens when you need related data for your objects.
Simple and generic model related data prefetch framework for Django solving the “1+N queries” problem that happens when you need related data for your objects.
In most of the cases you’ll have forward relations (foreign keys to something) and can use select_related to fetch that data on the same query. However, in some cases you cannot design your models that way and need data from reverse relations (models that have foreign keys to your objects).
Django has prefetch_related for this, however, this framework provides greater flexibility than Django’s prefetch_related queryset method at the cost of writting the mapping and query functions for the data. This has the advantage that you can do things prefetch_related cannot (see the latest_book example below).
Free software: BSD license
pip install django-prefetch
Use it as your model’s default manager (or as a base class if you have custom manager).
Python 2.7, 3.3+ or PyPy
Here’s a simple example of models and prefetch setup:
from django.db import models from prefetch import PrefetchManager, Prefetcher class Author(models.Model): name = models.CharField(max_length=100) objects = PrefetchManager( books = Prefetcher( filter = lambda ids: Book.objects.filter(author__in=ids), reverse_mapper = lambda book: [book.author_id], decorator = lambda author, books=(): setattr(author, 'books', books) ), latest_book = Prefetcher( filter = lambda ids: Book.objects.filter(author__in=ids), reverse_mapper = lambda book: [book.author_id], decorator = lambda author, books=(): setattr( author, 'latest_book', max(books, key=lambda book: book.created) if books else None ) ) ) class Book(models.Model): class Meta: get_latest_by = 'created' name = models.CharField(max_length=100) created = models.DateTimeField(auto_now_add=True) author = models.ForeignKey(Author)
Use it like this:
for a in Author.objects.prefetch('books', 'latest_book'): print a.books print a.latest_book
class LatestNBooks(Prefetcher): def __init__(self, count=2): self.count = count def filter(self, ids): return Book.objects.filter(author__in=ids) def reverse_mapper(self, book): return [book.author_id] def decorator(self, author, books=()): books = sorted(books, key=lambda book: book.created, reverse=True) setattr(author, 'latest_%s_books' % self.count, books[:self.count]) class Author(models.Model): name = models.CharField(max_length=100) objects = PrefetchManager( latest_n_books = LatestNBooks )
Use it like this:
from prefetch import P for a in Author.objects.prefetch(P('latest_n_books', count=5)): print a.latest_5_book
Check out the tests for more examples.
Document collect option of Prefetcher
Create tests covering custom collect and mapper
To run all the tests run:
Note, to combine the coverage data from all the tox environments run:
set PYTEST_ADDOPTS=--cov-append tox
Now support any primary key in master table with default mapper, not only “id” column. Contributed by Stas Fomin in #20.
Added support for Django 2.0
Fixed missing entry in changelog.
Added support for Django 1.11, dropped support for Django <1.9. Contributed by Martin Bachwerk in #16.
Fixed manager type check. Contributed by George Ma in #11.
Fixed issues with select_related being removed when prefetch is used (#9).
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Hashes for django_prefetch-1.2.3-py2.py3-none-any.whl