Skip to main content

Simple and generic model related data prefetch framework for Django solving the "1+N queries" problem that happens when you need related data for your objects.

Project description

Simple and generic model related data prefetch framework for Django solving the “1+N queries” problem that happens when you need related data for your objects.

In most of the cases you’ll have forward relations (foreign keys to something) and can use select_related to fetch that data on the same query. However, in some cases you cannot design your models that way and need data from reverse relations (models that have foreign keys to your objects).

Django has prefetch_related for this, however, this framework provides greater flexibility than Django’s prefetch_related queryset method at the cost of writting the mapping and query functions for the data. This has the advantage that you can do things prefetch_related cannot (see the latest_book example bellow).

  • Free software: BSD license

Installation guide

Install it:

pip install django-prefetch

Use it as your model’s default manager (or as a base class if you have custom manager).

Requirements

OS:Any
Runtime:Python 2.7, 3.3+ or PyPy
Packages:Django>=1.9

Example

Here’s a simple example of models and prefetch setup:

from django.db import models
from prefetch import PrefetchManager, Prefetcher

class Author(models.Model):
    name = models.CharField(max_length=100)

    objects = PrefetchManager(
        books = Prefetcher(
            filter = lambda ids: Book.objects.filter(author__in=ids),
            reverse_mapper = lambda book: [book.author_id],
            decorator = lambda author, books=(): setattr(author, 'books', books)
        ),
        latest_book = Prefetcher(
            filter = lambda ids: Book.objects.filter(author__in=ids),
            reverse_mapper = lambda book: [book.author_id],
            decorator = lambda author, books=(): setattr(
                author,
                'latest_book',
                max(books, key=lambda book: book.created)
            )
        )
    )

class Book(models.Model):
    class Meta:
        get_latest_by = 'created'

    name = models.CharField(max_length=100)
    created = models.DateTimeField(auto_now_add=True)
    author = models.ForeignKey(Author)

Use it like this:

for a in Author.objects.prefetch('books', 'latest_book'):
    print a.books
    print a.latest_book

Prefetcher arguments

Example models:

class LatestNBooks(Prefetcher):
    def __init__(self, count=2):
        self.count = count

    def filter(self, ids):
        return Book.objects.filter(author__in=ids)

    def reverse_mapper(self, book):
        return [book.author_id]

    def decorator(self, author, books=()):
        books = sorted(books, key=lambda book: book.created, reverse=True)
        setattr(author,
                'latest_%s_books' % self.count,
                books[:self.count])

class Author(models.Model):
    name = models.CharField(max_length=100)

    objects = PrefetchManager(
        latest_n_books = LatestNBooks
    )

Use it like this:

from prefetch import P

for a in Author.objects.prefetch(P('latest_n_books', count=5)):
    print a.latest_5_book

Note

P is optional and you can only use for prefetch definitions that are Prefetcher subclasses. You can’t use it with prefetcher-instance style definitions like in the first example. Don’t worry, if you do, you will get an exception explaining what’s wrong.

Other examples

Check out the tests for more examples.

TODO

  • Document collect option of Prefetcher
  • Create tests covering custom collect and mapper

Changelog

1.2.1 (2018-09-04)

  • Fixed missing entry in changelog.

1.2.0 (2018-09-04)

  • Added support for Django 1.11, dropped support for Django <1.9. Contributed by Martin Bachwerk in #16.

1.1.0 (2016-02-20)

  • Fixed a test assertion. Contributed by George Ma in #12.
  • Added support for Django 1.9. Contributed by Will Stott in #14.
  • Fixed use of deprecated field.rel.to momdel API (Django 1.9+).

1.0.1 (2015-09-05)

  • Fixed manager type check. Contributed by George Ma in #11.

1.0.0 (2014-12-05)

  • Fixed issues with select_related being removed when prefetch is used (#9).

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
django_prefetch-1.2.1-py2.py3-none-any.whl (5.8 kB) Copy SHA256 hash SHA256 Wheel py2.py3
django-prefetch-1.2.1.tar.gz (20.3 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page