django-allianceutils 4.1.0

Alliance Software common utilities for Django

Alliance Utils

A collection of utilities for django projects from Alliance Software.

• Installation
• Usage
  • API
  • Auth
  • Decorators
  • Management
    • Commands
    • Checks
  • Middleware
  • Migrations
  • Models
  • Rules
  • Serializers
  • Template
  • Template Tags
  • Tests
  • Util
• Changelog

Installation

pip install django-allianceutils

System Requirements

• Tested with django 4.2 and 5.0
  • Pull requests accepted for other versions, but at minimum we test against current LTS versions
• Python >=3.8

Usage

API

Mixins

SerializerOptInFieldsMixin

Regulates fields exposed on a Serializer by default & as requested based on query parameters or context.

• Pass 'include_fields' / 'opt_in_fields' thru query params or context to use.
• multiple fields can either be separated by comma eg, /?include_fields=first_name,email&opt_in_fields=gait_recognition_prediction
• or passed in the traditional list fashion eg, /?include_fields=first_name&include_fields=email&opt_in_fields=gait_recognition_prediction
• or mixed eg, /?include_fields=first_name,email&include_fields=boo
• By default, all "fields" defined in serializer, minus those listed in "opt_in_fields" would be returned.
• If "include_fields" is supplied, only fields requested this way would be returned.
• If "opt_in_fields" is supplied, fields requested this way PLUS fields from #1 or #2 would be returned.
• Pinned fields are always returned (defaults to primary key)

Usage:

class UserSerializer(SerializerOptInFieldsMixin, ModelSerializer):
    class Meta:
        model = User
        fields = (
            "id",
            "first_name",
            "last_name",
            "email",
            "region",
            "activated_at",
            "is_staff",
        )
        # These fields only returned if explicitly requested
        opt_in_only_fields = ["activated_at", "is_staff"]

Permissions

register_custom_permissions

• Creates permissions that have no model by linking them to an empty content type
• Django creates permissions as part of the post_migrate signal

Usage

def on_post_migrate(sender, **kwargs):
    register_custom_permissions("myapp", ("my_perm", "My Permission"))

SimpleDjangoObjectPermissions

Permission class for Django Rest Framework that adds support for object level permissions.

Differs from just DRF's DjangoObjectPermissions because it

• does not require a queryset
• uses the same permission for every request http method and ViewSet method

Notes

• The default django permissions system will always return False if given an object; you must be using another permissions backend
• As per DRF documentation: get_object() is only required if you want to implement object-level permissions
• WARNING If you override get_object() then you need to manually invoke self.check_object_permissions(self.request, obj)
• Will attempt to check permission both globally and on a per-object basis but considers it an error if the check returns True for both

Usage

• See DRF permissions policy for details on apply Permissions policies globally

• To apply to a specific view you need to set permission_required

class MyAPIView(SimpleDjangoObjectPermissions, APIView):
        permission_required = 'my_module.my_permission'
        permission_classes = [allianceutils.api.permissions.SimpleDjangoObjectPermissions]

If you have no object level permissions (eg. from rules) then it will just do a static permission check.

GenericDjangoViewsetPermissions

• Map viewset actions to Django permissions.
  • The model used for permission is extracted from the ViewSet
    • If you implement get_permission_model on the ViewSet that will be used
    • Otherwise it will call get_queryset on the ViewSet and extract the model from the returned queryset
• To alter this behaviour extends GenericDjangoViewsetPermissions and implement get_model
• Usage example:

class MyViewSet(GenericDjangoViewsetPermissions, viewsets.ModelViewSet):
    queryset = MyModel.objects.all()
    serializer_class = MySerializer

• GenericDjangoViewsetPermissions.default_actions_to_perms_map defines the default set of permissions. These can be extended or overridden using actions_to_perms_map:

class MyViewSet(GenericDjangoViewsetPermissions, viewsets.ModelViewSet):

    # ...

    actions_to_perms_map = {
        'create': []
    }

• No permissions will be required for the create action, but permissions for other actions will remain unchanged.
• By default permissions checks are passed the relevant model instance for per-object permission checks
  • This assumes that your backend doesn't ignore the model object (default django permissions simply ignore any object passed to a permissions check)
  • Since there is no model object, functions decorated with @list_route will pass None as the permissions check object

Parsers

CamelCaseJSONParser

Parser that recursively turns camelcase keys into underscored keys for JSON data. This can be set globally on the DEFAULT_PARSER_CLASSES setting or on a ViewSet on the parser_classes property.

CamelCaseMultiPartJSONParser

Parser that recursively turns camelcase keys into underscored keys for JSON data and handles file uploads. This parser supports receiving JSON data where a field value anywhere in the structure can be a file. This is achieved on the frontend by converting a structure like:

{
    name: 'Test',
    photo: File,
}

And converting it to

{
    name: 'Test',
    photo: '____ATTACHED_FILE_ID_1',
}

This is then set on a field jsonData and the file is set on ____ATTACHED_FILE_ID_1 and submitted as multipart. This parser then handles parsing the JSON data into a dict and setting each attached file on the correct key in the dict. Note that this works with nested data (ie. any File anywhere in a nested JSON structure is supported). To activate this behaviour the X-MultiPart-JSON header must be set to '1' or 'true'. If this header is not set it falls back to the default behaviour of MultiPartParser This can be set globally on the DEFAULT_PARSER_CLASSES setting or on a ViewSet on the parser_classes property. Example frontend code to activate:

let fileCount = 0;
const files = {};
const replacer = (key, value) => {
    if (value instanceof File) {
        const id = `____ATTACHED_FILE_ID_${fileCount++}`;
        files[id] = value;
        return id;
    }
    return value;
};
const stringifiedData = JSON.stringify(data, replacer);
const body = new FormData();
const body.append('jsonData', stringifiedData);
for (const [fileKey, file] of Object.entries(files)) {
    body.append(fileKey, file);
}
// eg. using a presto Endpoint
await myEndpoint.execute({
    body,
    headers: {
        // Remove default content type from endpoint (eg. json)
        'Content-Type': undefined,
        'X-MultiPart-JSON': true,
    },
});

Renderers

CamelCaseJSONRenderer

Renderer that recursively turns underscore-cased keys into camel-cased keys. This can be set globally on the DEFAULT_RENDERER_CLASSES setting or on a ViewSet on the renderer_classes property.

Auth

MinimalModelBackend

• allianceutils.auth.backends.MinimalModelBackend
  • Replaces the built-in django ModelBackend
  • Provides django model-based authentication
  • Removes the default authorization (permissions checks) except for checking is_superuser

ProfileModelBackend

• Backends for use with GenericUserProfile; see code examples there
• allianceutils.auth.backends.ProfileModelBackendMixin - in combo with AuthenticationMiddleware will set user profiles on request.user
  • If you want
    • allianceutils.auth.backends.ProfileModelBackend - convenience class combined with case insensitive username & default django permissions backend
      • this depended on authtools which appears to have been abandoned and does not work with django >= 3. If using django 3 then we recommended that you create your own backend in your app:

          class ProfileModelBackend(ProfileModelBackendMixin, MinimalModelBackend):
              # you'll need to implement case insensitivity either here or in the User Model
              pass
        

Permissions

NoDefaultPermissionsMeta

• Define a Meta class with empty default permissions so Django doesn't create any
• Usage example:

class User(GenericUserProfile):

    class Meta(NoDefaultPermissionsMeta):
        pass

PermissionNotImplementedError

• Subclass of NotImplementedError specific to permissions

identify_global_perms

• Takes a permission or list of permissions and splits them into global and object permissions, returning a tuple of (global permission list, object permission list). If the type can't be determined, the permission is returned in the global permission list.

AmbiguousGlobalPermissionWarning

• Raised if a permission cannot be classified as either global or per-object

reverse_if_probably_allowed

• Attempts to guess whether a user has permission to access a view to determine whether a URL should be displayed. Only for display purposes, not actual security, as it is not 100% reliable: can be used to, for example, hide the edit link in a CRUD view where the user does not have edit access. Takes the current request and the requested viewname, and optionally the specific object to be accessed.

Decorators

gzip_page_ajax

• Smarter version of django's gzip_page:
  • If settings.DEBUG not set, will always gzip
  • If settings.DEBUG set, will gzip only if request is an ajax request
• This allows you to use django-debug-toolbar in DEBUG mode (if you gzip a response then the debug toolbar middleware won't run)

Example

@allianceutils.views.decorators.gzip_page_ajax
def my_view(request: HttpRequest) -> httpResponse:
    data = {
        "message": "Hello World",
    }
    return django.http.JsonResponse(data)

method_cache

• Caches the results of a method on the object instance
• There is no thread synchronization so in some circumstances the method may be called multiple times if multiple threads share the object
• Only works for regular object methods with no arguments other than self.
  • Does not support @classmethod or @staticmethod
  • If you want more powerful caching behaviour then you can
    • use methodtools
    • wrap cachetools (examples here)
• Similar to @cached_property except that it works on methods instead of properties
• Differs from @lru_cache() in that
  • lru_cache uses a single cache for each decorated function
  • lru_cache will block garbage collection of values in the cache
  • A cache_clear() method is attached to the function but unlike lru_cache it is scoped to an object instance

Usage

class MyViewSet(ViewSet):

    # ...

    @method_cache
    def get_object(self):
        return super().get_object()

obj = MyViewSet()
obj.get_object() is obj.get_object()
obj.get_object.cache_clear()

Management

Commands

OptionalAppCommand

• A utility class that extends django.core.management.base.BaseCommand and adds optional argument(s) for django apps
• If app names are passed on the command line handle_app_config() will be called with the AppConfig for each app otherwise it will be called with every first-party app (as determined by isort)

Example:

class Command(allianceutils.management.commands.base.OptionalAppCommand):
    def add_arguments(self, parser):
        super().add_arguments(parser)
        parser.add_argument('--type', choices=('name', 'label'), default='name')

    def handle_app_config(self, app_config: AppConfig, **options):
        if options['type'] == 'name':
            print(f"Called with {app_config.name}")
        if options['type'] == 'label':
            print(f"Called with {app_config.label}")

print_logging

• Displays the current logging configuration in a hierarchical fashion
• Requires logging_tree to be installed

Checks

• Checks with no configuration are functions that can be passed directly to register
• Checks that expect parameters are classes that need to be instantiated

Setting up django hooks:

from django.apps import AppConfig
from django.core.checks import register
from django.core.checks import Tags

from allianceutils.checks import check_admins
from allianceutils.checks import check_db_constraints
from allianceutils.checks import check_duplicated_middleware
from allianceutils.checks import CheckExplicitTableNames
from allianceutils.checks import check_git_hooks
from allianceutils.checks import CheckReversibleFieldNames
from allianceutils.checks import CheckUrlTrailingSlash

class MyAppConfig(AppConfig):
    # ...

    def ready(self):
        register(check=check_admins, tags=Tags.admin, deploy=True)
        register(check=check_db_constraints, tags=Tags.database)
        register(check=check_duplicated_middleware, tags=Tags.admin)
        register(check=CheckExplicitTableNames(), tags=Tags.models)
        register(check=check_git_hooks, tags=Tags.admin)
        register(check=CheckReversibleFieldNames(), tags=Tags.models)
        register(check=CheckUrlTrailingSlash(expect_trailing_slash=True), tags=Tags.url)

CheckUrlTrailingSlash

• Checks that your URLs are consistent with the settings.APPEND_SLASH
• Arguments:
  • ignore_attrs - skip checks on url patterns where an attribute of the pattern matches something in here (see example above)
    • Most relevant attributes of a RegexURLResolver:
      • _regex - string used for regex matching. Defaults to [r'^$']
      • app_name - app name (only works for include() patterns). Defaults to ['djdt'] (django debug toolbar)
      • namespace - pattern defines a namespace
      • lookup_str - string defining view to use. Defaults to ['django.views.static.serve']
    • Note that if you skip a resolver it will also skip checks on everything inside that resolver
• Note: If using Django REST Framework's DefaultRouter then you need to turn off include_format_suffixes:

router = routers.DefaultRouter(trailing_slash=True)
router.include_format_suffixes = False
router.register(r'myurl', MyViewSet)
urlpatterns += router.urls

check_admins

• Checks that settings.ADMINS has been properly set in settings files.

check_git_hooks

• Checks that git hookshave been set up, one of:
  • .git/hooks directory has been symlinked to the project's git-hooks
  • husky hooks have been installed

check_db_constraints

• Checks that all models that specify db_constraints in their Meta will generate unique constraint names when truncated by the database.

CheckExplicitTableNames

• Checks that all first-party models have db_table explicitly defined on their Meta class, and the table name is in lowercase
• Arguments:
  • enforce_lowercase - check that there are no uppercase characters in the table name
  • ignore_labels - if an app label (eg silk) or app_label + model labels (eg silk.request) matches something in ignore_labels then it will be ignored.
    • allianceutils.checks.DEFAULT_TABLE_NAME_CHECK_IGNORE contains a default list of apps/models to ignore
    • Can be either a str or a regex (anything that contains a .match() method)

CheckReversibleFieldNames

• Checks that all models have fields names that are reversible with underscorize/camelize/camel_to_underscore/underscore_to_camel
• Arguments:
  • ignore_labels - ignore these apps/models: see CheckExplicitTableNames

Middleware

HttpAuthMiddleware

• Middleware to enable basic http auth to block unwanted traffic from search engines and random visitors

  • Intended to be used on dev / staging servers
  • Is not a full authorization system: is a single hardcoded username/password and should be used on top of a proper authorization system

• Setup

  • Add allianceutils.middleware.HttpAuthMiddleware to MIDDLEWARE.
  • Add HTTP_AUTH_USERNAME and HTTP_AUTH_PASSWORD to appropriate setting file, e.g. settings/production_staging.py
    • Remember that you shouldn't be hardcoding credentials in code: read content from env vars or file

CurrentUserMiddleware

• Middleware to enable accessing the currently logged-in user without a request object.

  • Assumes that threading.local is not shared between requests (an assumption also made by django internationalisation)

• Setup

  • Add allianceutils.middleware.CurrentUserMiddleware to MIDDLEWARE.

• Usage

from allianceutils.middleware import CurrentUserMiddleware

user = CurrentUserMiddleware.get_user()

QueryCountMiddleware

• Warns if query count reaches a given threshold

  • Threshold can be changed by setting settings.QUERY_COUNT_WARNING_THRESHOLD

• Usage

  • Add allianceutils.middleware.CurrentUserMiddleware to MIDDLEWARE.
  • Uses the warnings module to raise a warning; by default this is suppressed by django
    • To ensure QueryCountWarning is never suppressed

warnings.simplefilter('always', allianceutils.middleware.QueryCountWarning)

• To increase the query count limit for one request, you can call QueryCountMiddleware.increase_threshold(request, increment)
• To set the query count limit for one request you can call QueryCountMiddleware.set_threshold(request, threshold)
  • Rather than hardcode a new limit, increase_threshold() is generally preferable
  • This can be useful to disable checks entirely (pass 0 as the new limit)

def my_view(request, *args, **kwargs):
    request.QUERY_COUNT_WARNING_THRESHOLD += 10
    ...

Migrations

Run SQL function

• Wrapper to RunSQL that reads SQL from a file instead of inline in python
• The reason you would do this as an external file & function is so that squashed migrations don't become unwieldy (django will inline and strip whitespace in the SQL)

Usage:

class Migration(migrations.Migration):
    # ...
    operations = [
        allianceutils.migrations.RunSQLFromFile('my_app', '0001_intial.sql'),
    ]

Models

Utility functions / classes

combine_querysets_as_manager

• allianceutils.models.combine_querysets_as_manager(Iterable[Queryset]) -> Manager
• Replacement for django_permanent.managers.MultiPassThroughManager which no longer works in django 1.8
• Returns a new Manager instance that passes through calls to multiple underlying queryset_classes via inheritance

NoDeleteModel

• A model that blocks deletes in django
  • Can still be deleted with manual queries
• Read django docs about manager inheritance
  • If you wish add your own manager, you need to combine the querysets:

class MyModel(NoDeleteModel):
        objects = combine_querysets_as_manager(NoDeleteQuerySet, MyQuerySet)

GenericUserProfile

Allows you to iterate over a User table and have it return a corresponding Profile record without generating extra queries

Minimal example:

# ------------------------------------------------------------------
# base User model

# If you're using django auth instead of authtools, you can just use
# GenericUserProfileManager instead of having to make your own manager class
class UserManager(GenericUserProfileManagerMixin, authtools.models.UserManager):
    pass

class User(GenericUserProfile, authtools.models.AbstractEmailUser):
    objects = UserManager()
    profiles = UserManager(select_related_profiles=True)

    # these are the tables that should be select_related()/prefetch_related()
    # to minimise queries
    related_profile_tables = [
        'customerprofile',
        'adminprofile',
    ]

    # the default implementation will iterate through the related profile tables
    # and return the first profile it can find. If you have custom logic for
    # choosing the profile for a user then you can do that here
    #
    # You would normally not access this directly but instead use the`.profile`
    # property that caches the return value of `get_profile()` and works
    # correctly for both user and profile records
    def get_profile(self) -> Model:
        # custom logic
        if datetime.now() > datetime.date(2000,1,1):
            return self
        return super().get_profile()


# ------------------------------------------------------------------
# Custom user profiles
class CustomerProfile(User):
    customer_details = models.CharField(max_length=191)


class AdminProfile(User):
    admin_details = models.CharField(max_length=191)

# ------------------------------------------------------------------
# Usage:

# a list of User records
users = list(User.objects.all())

# a list of Profile records: 1 query
# If a user has no profile then you get the original User record back
profiles = list(User.profiles.all())

# we can explicitly perform the transform on the queryset
profiles = list(User.objects.select_related_profiles().all())

# joining to profile tables: 1 query
# This assumes that RetailLocation.company.manager is a FK ref to the user table
# The syntax is a bit different because we can't modify the query generation
# in an unrelated table
qs = RetailLocation.objects.all()
qs = User.objects.select_related_profiles(qs, 'company__manager')
location_managers = list((loc, loc.company.manager.profile) for loc in qs.all())

• There is also an authentication backend that will load profiles instead of just User records
• If the User model has no get_profile() method then this backend is equivalent to the built-in django django.contrib.auth.backends.ModelBackend

# ------------------------------------------------------------------
# Profile authentication middleware
AUTH_USER_MODEL = 'my_site.User'
AUTHENTICATION_BACKENDS = [
    'allianceutils.auth.backends.ProfileModelBackend',
]


def my_view(request):
    # standard django AuthenticationMiddleware will call the authentication backend
    profile = request.user
    return HttpResponse('Current user is ' + profile.username)

• Limitations:
  • Profile iteration does not work with .values() or .values_list()

raise_validation_errors

• The raise_validation_errors context manager enables cleaner code for constructing validation
  • Django documentation recommends raising a ValidationError when you encounter a problem
  • This creates a poor user experience if there are multiple errors: the user only sees the first error and has to resubmit a form multiple times to fix problems
• raise_validation_errors accepts an (optional) function to wrap
  • The context manager returns a ValidationError subclass with an add_error function that follows the same rules as django.forms.forms.BaseForm.add_error
  • If the wrapped function raises a ValidationError then this will be merged into the ValidationError returned by the context manager
  • If the wrapped function raises any other exception then this will not be intercepted and the context block will not be executed
  • At the end of a block,
    • If code in the context block raised an exception (including a ValidationError) then this will not be caught
    • If ValidationError the context manager returned has any errors (either from ve.add_error() or from the wrapped function) then this will be raised

    def clean(self):
        with allianceutils.models.raise_validation_errors(super().clean) as ve:
            if some_condition:
                ve.add_error(None, 'model error message')
            if other_condition:
                ve.add_error('fieldname', 'field-specific error message')
            if other_condition:
                ve.add_error(None, {
                    'fieldname1': field-specific error message',
                    'fieldname2': field-specific error message',
                })
            if something_bad:
                raise RuntimeError('Oh no!')

            # at the end of the context, ve will be raised if it contains any errors
            #   - unless an exception was raised in the block (RuntimeError example above) in which case
            #     the raised exception will take precedence

• Sometimes you already have functions that may raise a ValidationError and add_error() will not help
  • The capture_validation_error() context manager solves this problem
  • Note that due to the way context managers work, each potential ValidationError needs its own with capture_validation_error context

    def clean(self):
        with allianceutils.models.raise_validation_errors() as ve:
             with ve.capture_validation_error():
                 self.func1()
             with ve.capture_validation_error():
                 self.func2()
             with ve.capture_validation_error():
                 raise ValidationError('bad things')
            # all raised ValidationErrors will be collected, merged and raised at the end of this block

Rules

• Utility functions that return predicates for use with django-rules

from allianceutils.rules import has_any_perms, has_perms, has_perm

# requires at least 1 listed permission
rules.add_perm('northwind.publish_book', has_any_perms('northwind.is_book_author', 'northwind.is_book_editor'))

# requires listed permission
rules.add_perm('northwind.unpublish_book', has_perm('northwind.is_book_editor'))

# requires all listed permissions
rules.add_perm('northwind.sublicense_book', has_perms('northwind.is_book_editor', 'northwind.can_sign_contracts'))

Serializers

JSON Ordered

• A version of django's core json serializer that outputs field in sorted order

• The built-in one uses a standard dict with completely unpredictable order which causes json diffs to show spurious changes

• Setup

  • Add to SERIALIZATION_MODULES in your settings
  • This will allow you to do fixture dumps with --format json_ordered
  • Note that django treats this as the file extension to use

SERIALIZATION_MODULES = {
    'json_ordered': 'allianceutils.serializers.json_ordered',
}

Template

resolve

• Resolves different types of kwarg values consistently.

  • If value is a NodeList then it is rendered
  • If value is a FilterExpression then it is resolved
  • If value is a dict then each element is resolved if it is a FilterExpression, or else returned as is
  • Otherwise value is returned as is

• Example Usage

    def resolved_kwargs(self, context: Context):
        pk = resolve(self.pk, context)
        model = resolve(self.model_name, context)
        object = resolve(self.object, context)

token_kwargs

• Re-implements the token_kwargs function from the Django base template library. This allows expanding the range of possible keywords to include '-' for aria attributes (e.g. aria-label="My Label"), and ':' for namespaced attributes (e.g. xlink:href="foo"). Used internally in parse_tag_arguments - see that section for usage

parse_tag_arguments

• Implements a stripped-down version of django.template.library.parse_bits() to parse tokens passed to tags in Django templates.

  • Takes the parser to process the tag, the token passed to the tag, and the kwarg supports_as
  • Returns a tuple of (args: list, kwargs: dict, target_var: str):
    • args: a list of args as FilterExpressions
    • kwargs: a dict of kwargs
    • target_var: if supports_as is True and as <variable> is specified, returns the string reference for the tag. Otherwise returns None

• Example Usage

class StylesheetNode(template.Node, BundlerAsset):
    def __init__(self, filename: Path, origin: Origin | None, attrs=None, target_var=None):
        self.filename = filename
        self.attrs = attrs
        self.target_var = target_var
        super().__init__(origin or Origin(UNKNOWN_SOURCE))

    def render(self, context: Context):
        if self.target_var:
            context[self.target_var] = get_classes(self.filename)

@register.tag("stylesheet")
def stylesheet(parser: template.base.Parser, token: template.base.Token):
    args, kwargs, target_var = parse_tag_arguments(parser, token, supports_as=True)
    filename = args[0]
    return StylesheetNode(filename, parser.origin, kwargs, target_var=target_var)

    {% stylesheet "./theme.css" as styles %}

    <div class="{{ styles.section }}">
        <h1 class="{{ styles.heading }}">My View</h1>
        ...
    </div>

build_html_attrs

• Takes a dict of HTML tag attributes and transforms values into escaped strings suitable for use as HTML tag attributes. Can also pass a list of prohibited_attrs to prevent passing attributes which should not be passed to template tags.

• Example Usage

from django.utils.html import format_html
...
class LinkNode(template.Node):
    def __init__(
        self,
        *,
        href: FilterExpression | str | None = None,
        **extra_kwargs,
    ) -> None:
        self.href = href

    ...

    def render(self, context: Context) -> SafeString:
        href = resolve(self.href, context)
        html_kwargs = { "href": href }
        html_attrs = build_html_attrs(html_kwargs)
        return format_html(
            "<a {}>{}</a>",
            html_attrs,
        )

    {% link "/home" %}

is_static_expression

• Checks if a given FilterExpression is static using the same method as Django's resolve implementation for the Variable class

• Example Usage

def validate_tag(parser, token):
    tag_name = token.split_contents()[0]
    args, _, _ = parse_tag_arguments(parser, token)

    for arg in args:
        if not is_static_expression(arg):
            raise TemplateSyntaxError(
                f"{tag_name} must be passed static strings for its arguments (encountered variable '{arg.var}')"
            )

Template Tags

render_entry_point

• Replaces old usage of django-webpack-loader

  • At time of writing django-webpack-loader does not have a stable release that works with webpack 4
  • Worked at bundle level rather than entry point. See below for how we embed tags based on entry point.

• Reads JSON files generated by EntryPointBundleTracker and embeds the required bundles in the page

  • Will output tags for all resources of the specified type. eg. Given JSON structure of:

   {
     "status": "done",
     "entrypoints": {
       "admin": [
         {
           "name": "runtime.bundle.js",
           "contentHash": "e2b781da02d36dad3aff"
         },
         {
           "name": "common.bundle.js",
           "contentHash": "639269b921c8cf869c5f"
         },
         {
           "name": "common.bundle.css",
           "contentHash": "d60a0fa36613ea58a23d"
         },
         {
           "name": "admin.bundle.js",
           "contentHash": "c78fb252d4e00207afef"
         }
       ]
     },
     "publicPath": "/assets/"
   }
  

  Output for {% render_entry_point 'admin' 'js' %}:

  <script type="text/javascript" src="/assets/runtime.bundle.js?e2b781da02d36dad3aff"></script>
  <script type="text/javascript" src="/assets/common.bundle.js?639269b921c8cf869c5f"></script>
  <script type="text/javascript" src="/assets/admin.bundle.js?c78fb252d4e00207afef"></script>
  

  Output for {% render_entry_point 'admin' 'css' %}:

  <link type="text/css" href="/assets/common.bundle.css?d60a0fa36613ea58a23d" rel="stylesheet" />
  

• As an entry point maps to a single HTML file it's expected you would only use this tag for a single entry point on a page but generally would call it for both js and css

• Arguments

  • entry_point_name - Name of the entry point. This should match one of the entries to 'entry' in the webpack config.
  • resource_type - The resource type to embed; either js or css.

• Optional Arguments

  • attrs - String representing extra attributes to pass to the HTML tag
  • config='DEFAULT' - String index into the settings WEBPACK_LOADER dict. Defaults to 'DEFAULT'.

• Config

  • Configuration can be specified via the WEBPACK_LOADER setting. This is a dict indexed by the config name (defaults to 'DEFAULT')
  • Options
    • STATS_FILE - the path to the stats file to read
    • INCLUDE_QUERY_HASH - whether to include the content hash in the query string. Defaults to true.
    • BASE_URL - a URL to prepend to all chunks when rendered. This can be used when files are stored on a different host (eg. CDN).

• Example Usage

{% load alliance_webpack %}
<html>
<head>
  {% render_entry_point 'app' 'css' %}
</head>
<body>

  ...

  {% if DEBUG %}
    {# See https://reactjs.org/docs/cross-origin-errors.html #}
    {% render_entry_point entry_point 'js' attrs="crossorigin" %}
  {% else %}
    {% render_entry_point entry_point 'js' %}
  {% endif %}
</body>
</html>

default_value

• Sets default value(s) on the context in a template
• This is useful because some built-in django templates raise warnings about unset variables (eg is_popup in the django admin template)
• Note that some tags (eg with) save & restore the context state; if done inside such a template tag default_value will not persist when the state is restored

{% load default_value %}
{{ default_value myvar1=99 myvar2=myvar1|upper }}
{{ myvar1 }} {{ myvar2 }}

Tests

suppress_silk

• Decorator to disable silk SQL query logging.

• This is needed for tests that use assertNumQueries() since otherwise the query count may include silk's EXPLAIN

• Example Usage

    @suppress_silk()
    def test_lookup_object(self):
        """Test that a query is run if object is not provided in template"""
        user = self.get_privileged_user()
        viewname = self.VIEW_KWARGS
        view_kwargs = {"pk": user.pk}

        request = HttpRequest()

        # Without suppress_silk, an EXPLAIN query will be added
        with self.assertNumQueries(1):
            tpl = Template('{% load link %}{% link "' + viewname + '" pk=pk %}foo{% endlink %}')
            tpl.render(Context({"request": request, "user": user, **view_kwargs}))

logging_filter

• Decorator to disable logging for specified log names

• This is useful because using override_settings(LOGGING=...) triggers an update to the python logging settings

• Example Usage

    @logging_filter(["django.request"])
    def test_django_validation_errors(self):
        url = reverse("django-validation-test-url")
        for key, expect in EXPECTED_RESPONSES.items():
            with self.subTest(key=key):
                response = self.client.post(url, data={"error_key": key}, format="json")
                self.assertEqual(400, response.status_code)
                self.assertEqual(expect, response.json())

warning_filter

• Apply a warning.simplefilter() for the specified warnings

• Example Usage

    @warning_filter("ignore", category=DeprecationWarning)
    def test_relying_on_deprecated_feature(self):
        instance = MyModel.objects.filter(deprecated_filter=True)
        assertIsInstance(instance, MyModel)

Util

add_autoreload_extra_files

• Adds files to the autoreloader watch list
  • Works with both the built-in runserver and runserver_plus from django-extensions
  • If DEBUG is not enabled then this will do nothing
  • This should be called from inside the ready() method of an AppConfig

class MyAppConfig(AppConfig):
    def ready(self):
        extra_files = [
          "/data/file.csv",
        ]
        add_autoreload_extra_files(extra_files)

camelize

• Better version of djangorestframework-camel-case
  • DRF-CC camel cases every single key in the data tree.
• These functions allow you to indicate that certain keys are data, not field names

tree = {
    "first_name": "Mary",
    "last_name": "Smith",
    "servers": {
        "server_west.mycompany.com": {
            'user_accounts': {
                'mary_smith': {
                    'home_dir': '/home/mary_smith',
                },
            },
        },
    },
}
# the keys at tree['servers'] and tree['servers']['serve_west.mycompany.com']['user_accounts'] will not be camel cased
output_tree = allianceutils.util.camelize(tree, ignore=['servers', 'servers.*.user_accounts'])
output_tree == {
    "firstName": "Mary",
    "lastName": "Smith",
    "servers": {
        "server_west.mycompany.com": {
            'userAccounts': {
                'mary_smith': {
                    'home_dir': '/home/mary_smith',
                },
            },
        },
    },
}

• allianceutils.util.camelize(data, ignores) - underscore case => camel case a json tree of data
• allianceutils.util.underscorize(data, ignores) - camel case => underscore case a json tree of data
• allianceutils.util.camel_to_underscore(str) - underscore case => camel case a string
• allianceutils.util.underscore_to_camel(str) - camel case => underscore case a string
• It is assumed that words will not begin with numbers:
  • zoo_foo99_bar is okay
  • zoo_foo_99bar will result in an irreversible transformation (zooFoo99bar => zoo_foo99_bar)

get_firstparty_apps

util.get_firstparty_apps can be used to retrieve app_configs considered to be first party, ie, all that does not come from a third party package. This is beneficial when you want to write your own checks by excluding things you dont really care - a sample usage can be found inside 'checks.py', or used as such:

from allianceutils.util import get_firstparty_apps

app_configs = get_firstparty_apps()
models_to_be_checked = {}

for app_config in app_configs:
    models_to_be_checked.update({
        model._meta.label: model
        for model
        in app_config.get_models()
    })

python_to_django_date_format

• Converts a python strftime/strptime datetime format string into a django template/PHP date format string
• Codes with no equivalent will be dropped

Example:

allianceutils.util.date.python_to_django_date_format("%Y%m%d %H%M%S")
# returns "Ymd His"

retry_fn

• Repeatedly (up to a hard limit) call specified function while it raises specified exception types or until it returns

from allianceutils.util import retry_fn

# Generate next number in sequence for a model
# Number here has unique constraint resulting in IntegrityError being thrown
# whenever a duplicate is added. can be useful for working around mysql's lack
# of sequences
def generate_number():
    qs = MyModel.objects.aggregate(last_number=Max(F('card_number')))
    next_number = (qs.get('last_card_number') or 0) + 1
    self.card_number = card_number
    super().save(*args, **kwargs)
retry_fn(generate_number, (IntegrityError, ), 10)

Experimental

• These are experimental and may change without notice
  • document_reverse_accessors management command

Changelog

See CHANGELOG.md

Development

• To create a clean local environment
  • python3 -m venv venv && source venv/bin/activate && pip install --upgrade pip
  • poetry install --no-root --sync --only=main --extras=""
  • Note that due to a poetry bug extras are currently not removed
  • This will install the latest django version; if you want to test a specific django version you need to pip install it manually
• Dev dependencies
  • poetry install --no-root --sync --with=dev --extras "extras mysql postgres"

Release Process

Poetry Config

• Add test repository
  • poetry config repositories.testpypi https://test.pypi.org/legacy/
  • Generate an account API token at https://test.pypi.org/manage/account/token/
  • poetry config pypi-token.testpypi ${TOKEN}
    • On macs this will be stored in the login keychain at poetry-repository-testpypi
• Main pypi repository
  • Generate an account API token at https://pypi.org/manage/account/token/
  • poetry config pypi-token.pypi ${TOKEN}
    • On macs this will be stored in the login keychain at poetry-repository-pypi

Publishing a New Release

* Update CHANGELOG.md with details of changes and new version
* Run `bin/build.py`. This will extract version from CHANGELOG.md, bump version in `pyproject.toml` and generate a build for publishing
* Commit version bumps
* Tag with new version and update the version branch:
    * `ver=$( poetry version --short ) && echo "Version: $ver"`
    * `git tag v/$ver`
    * `git push --tags`
* To publish to test.pypi.org
    * `poetry publish --repository testpypi`
* To publish to pypi.org
    * `poetry publish`

Testing

• To run test cases
  • The django settings module is test_allianceutils/settings.py
    • The following env vars are optional but you may want to set them if the default don't match your local setup:
      • DB_NAME
      • DB_HOST
      • DB_PORT
      • DB_USER
      • DB_PASSWORD
• tox
  • used to run tests against different django/python/database versions
  • tox to run all tests. Will require that you have a postgres & mysql server running.
    • tox -f django42 will run the subset of tests that cover django 4.2. Check tox.ini for the list of tested environments.
• When you push to github a github Actions workflow will be triggered (see .github/workflows/django.yml)