django-shouty-orm 0.1.1

Applies a monkeypatch which forces Django's ORM to error far more loudly in certain cases

Author:

Keryn Knight

Version:

0.1.1

Rationale

I want to use MyModel.objects.only() and .defer() because that’s the correct thing to do, even if it’s not the default thing Django does. But using .only() and .defer() in Django is an absolute footgun because any attempt to subsequently access the missing fields will … do another query

Similarly, I don’t want Django to silently allow me to do N+1 queries for related managers/querysets. But it does, so there’s another footgun.

This module then, is my terrible attempt to fix the footguns automatically, by forcing them to raise exceptions where possible, rather than do the query. This flies in the face of some other proposed solutions over the years on the mailing list, such as automatically doing prefetch_related or select_related.

I think/hope that the package pairs well with django-shouty-templates to try and surface some of the small pains I’ve had over the years.

What it does

All of the following examples should raise an exception because they pose a probable additional +1 (or more) queries.

Accessing fields intentionally not selected:

>>> u = User.objects.only('pk').get(pk=1)
>>> u.username
MissingLocalField("Access to username [...]")
>>> u = User.objects.defer('username').get(pk=1)
>>> u.email
>>> u.username
MissingLocalField("Access to username [...]")

Access to relationships that have not been selected:

>>> le = LogEntry.objects.get(pk=1)
>>> le.action_flag
>>> le.user.pk
MissingRelationField("Access to user [...]")

Access to reverse relationships that have not been selected:

>>> u = User.objects.only('pk').get(pk=1)
>>> u.logentry_set.all()
MissingReverseRelationField("Access to logentry_set [...]")

Pretty much all relationship access (normal or reverse, OneToOne or ForeignKey or ManyToMany) should be blocked unless select_related or prefetch_related were used to include them.

Setup

Add shoutyorm or shoutyorm.Shout to your settings.INSTALLED_APPS

I’d certainly suggest that you should only enable it when DEBUG is True or during your test suite.

Dependencies

• Django 2.2+ (obviously)

• wrapt 1.11+ (for proxying managers/querysets transparently)

Optional configuration

• settings.SHOUTY_LOCAL_FIELDS may be True|False

  Accessing fields which have been deferred via .only() and .defer() at the QuerySet level will error loudly.

• settings.SHOUTY_RELATION_FIELDS may be True|False

  Accessing OneToOnes which have not been .select_related() at the QuerySet level will error loudly. Accessing local foreignkeys which have not been prefetch_related() or select_related() at the queryset level will error loudly.

• settings.SHOUTY_RELATION_REVERSE_FIELDS may be True|False

  Accessing foreignkeys from the “other” side (that is, via the reverse relation manager) which have not been .prefetch_related() at the QuerySet level will error loudly.

Tests

Just run python3 -m shoutyorm and hope for the best. I usually do.

Alternatives

A similar similar approach is taken by django-seal but without the onus/burden of subclassing from specific models. I’ve not looked at the implementation details of how seal works, but I expect I could’ve saved myself quite a lot of headache by seeing what steps it takes in what circumstances, rather than constantly hitting breakpoints and inspecting state.

A novel idea is presented in django-eraserhead of specifically calling out when you might be able to use defer() and only() to reduce your selections, but introducing those optimisations still poses a danger of regression without a test suite and this module.

Having started writing this list of alternatives, I am reminded of nplusone and it turns out that has Django support and a setting for raising exceptions… So all of this patch may be moot, because I expect that covers a lot more? Again I’ve not looked at their implementation but I’m sure it’s miles better than this abomination.

The license

It’s FreeBSD. There’s should be a LICENSE file in the root of the repository, and in any archives.

---

Copyright (c) 2020, Keryn Knight All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

---

Change history for django-shouty-orm

0.1.1

• Initial release