django-govuk-notify-outbox 2.0.0

django-govuk-notify-outbox

A reusable Django app to send emails via GOV.UK notify robustly and with good observability.

Contents

• Features
• Prerequisites
• Installation
• Usage
  • Fire and forget
  • Attaching to your own models
  • The 'reference' as an idempotency key to avoid duplicate emails
  • Throttling
• Email status descriptions
  • Happy path
  • Happy and unhappy paths
• Configuration
• Architecture
  • Why the sync task?
  • Why background tasks at all?
  • Why store data at all?
• License

Features

• Sends emails outside of the request/response cycle (via Celery)
• Automatically retries communication with GOV.UK Notify up to a maximum number / maximum amount of time
• Avoids duplicating emails, even in the case of unfortunately timed downtime
• Usage is consistent with the send_email_notification function from notifications-python-client, with an exception to how the reference field is handled to avoid duplicating emails
• Emails and updates from GOV.UK Notify (including HTTP errors and Python exceptions) are stored in Django models...
• ... so can be linked (via foreign key or many-to-many relation) to other models to be used in the user-facing application
• ... and are visible in the Django admin interface

Prerequisites

• An existing (even if newly created) Django project
• ... which is configured with Celery and Celery Beat to run background tasks
• A GOV.UK Notify service API Key

Installation

1. Install django-govuk-notify-outbox from PyPI.

   Typically you would do this by adding django-govuk-notify-outbox as a dependency to your existing Django project, for example with uv:

   uv add django-govuk-notify-outbox
   

2. Add django_govuk_notify_outbox to settings.INSTALLED_APPS

   INSTALLED_APPS = [
       # ...
       "django_govuk_notify_outbox",
   ]
   

3. Add settings.DJANGO_GOVUK_NOTIFY_OUTBOX__API_KEY with your GOV.UK Notify API key, typically from an environment variable.

   import os
   
   DJANGO_GOVUK_NOTIFY_OUTBOX__API_KEY = os.environ.get('DJANGO_GOVUK_NOTIFY_OUTBOX__API_KEY')
   

4. Add django_govuk_notify_outbox.tasks.sync to settings.CELERY_BEAT_SCHEDULE on a frequent schedule, for example every 5 seconds. If there is nothing to do, this will finish quickly and without connecting to GOV.UK notify.

   CELERY_BEAT_SCHEDULE = {
       "sync": {
           "task": "django_govuk_notify_outbox.tasks.sync",
           "schedule": 5,
       },
   }
   

5. Run python manage.py migrate to create the models. Often projects are setup to do this automatically on startup.

Usage

Fire and forget

Once installed, emails can be sent from any part of your Django project with the django_govuk_notify_outbox.utils.send_email_notification function, which is designed to be consistent with the "fire and forget"-style send_email_notification function from notifications-python-client

from django_govuk_notify_outbox.utils import send_email_notification

send_email_notification(
    email_address="name@example.invalid",
    template_id="9d751e0e-f929-4891-82a1-a3e1c3c18ee3",
)

Attaching to your own models

The return value of send_email_notification is an instance of the Django model django_govuk_notify_outbox.models.EmailMessage, and it can be attached to your own models in the usual way via ForeignKey or ManyToManyField. Each EmailMessage is updated through the email sending process until is is delivered or failed.

For example:

# In your app's models.py
from django.db import models
from django_govuk_notify_outbox.models import EmailMessage

class MyModel(models.Model):
    email_messages = models.ManyToManyField(EmailMessage)

# Anywhere in your app
from django_govuk_notify_outbox.utils import send_email_notification

email_message = send_email_notification(
    email_address="name@example.invalid",
    template_id="9d751e0e-f929-4891-82a1-a3e1c3c18ee3",
)
my_model_instance = MyModel.create()
my_model_instance.email_messages.add(email_message)
print(email_message.status)  # See below

The 'reference' as an idempotency key to avoid duplicate emails

The reference argument to send_email_notification is appended with a unique key before sending it to GOV.UK Notify to turn it into an idempotency key. The allows safe retries: no matter what happens in terms of downtime and retries, emails will not be duplicated.

The full reference according to GOV.UK Notify is stored in the reference_with_idempotency_key field on each EmailMessage instance.

There is currently no way to disable this behaviour.

Throttling

There is one extension to send_email_notification from notifications-python-client: the ability to throttle emails sent to each email address. This done by passing a throttle_max_per_minute argument that specifies the maximum number of emails that can be send to the target address per minute, as counted in the previous 60 seconds.

from django.db import transaction
from django_govuk_notify_outbox.utils import send_email_notification

with transaction.atomic():
    send_email_notification(
        email_address="name@example.invalid",
        template_id="9d751e0e-f929-4891-82a1-a3e1c3c18ee3",
        throttle_max_per_minute=4,
    )

If a call is throttled, the email message is not saved to the database (and so not sent) and a django_govuk_notify_outbox.ThrottledError exception is raised by the call to send_email_notification.

gmail.com and googlemail.com email addresses are normalised for the purposes of throttling:

• They are both treated as being sent to gmail.com
• Dots in the local / user part are removed
• Everything from the first '+' onwards in the local / user part is removed

To use throttle_max_per_minute the call to send_email_notification must be wrapped in a database transaction, and a PostgreSQL transaction-level advisory lock is acquired under the hood for the duration of the transaction. The lock has an ID equal to the first 8 bytes of the SHA256 of the normalised email address. This means that throttling is only available if using PostgreSQL.

Email status descriptions

Each EmailMessage.status field can be equal to one of 10 possible values. For the value that reflect a GOV.UK Notify status, see https://docs.notifications.service.gov.uk/python.html#email-status-descriptions for details.

Happy path

5 statuses make up the happy path when there are no problems sending an email:

stateDiagram-v2

    sendingToNotify: sending-to-notify
    sentToNotify: sent-to-notify
    notifyCreated: notify-created
    notifySending: notify-sending
    notifyDelivered: notify-delivered

    [*] --> sendingToNotify
    sendingToNotify --> sentToNotify
    sentToNotify --> notifyCreated
    notifyCreated --> notifySending
    notifySending --> notifyDelivered
    notifyDelivered --> [*]

Happy and unhappy paths

How EmailMessage.status can change in all cases of both success and failure is shown in the following diagram:

stateDiagram-v2

    sendingToNotify: sending-to-notify
    retryingSendToNotify: retrying-send-to-notify
    sentToNotify: sent-to-notify
    failedSendToNotify: failed-send-to-notify
    notifyCreated: notify-created
    notifySending: notify-sending
    notifyDelivered: notify-delivered
    notifyPermanentFailure: notify-permanent-failure
    notifyTemporaryFailure: notify-temporary-failure
    notifyTechnicalFailure: notify-technical-failure

    [*] --> sendingToNotify

    sendingToNotify --> retryingSendToNotify
    retryingSendToNotify --> failedSendToNotify
    sendingToNotify --> sentToNotify
    retryingSendToNotify --> sentToNotify

    failedSendToNotify --> [*]

    sentToNotify --> notifyCreated
    notifyCreated --> notifySending

    notifySending --> notifyDelivered
    notifySending --> notifyPermanentFailure
    notifySending --> notifyTemporaryFailure
    notifySending --> notifyTechnicalFailure

    notifyPermanentFailure --> [*]
    notifyTemporaryFailure --> [*]
    notifyTechnicalFailure --> [*]

    notifyDelivered --> [*]

Configuration

Configuration is via settings that go into your Django project's settings.py

Architecture

Why the sync task?

The sync task does two things. It retries sending to GOV.UK Notify, and it fetches updates on sent emails. There are are three reasons for this:

1. It is the most robust way to make sure retries are attempted, especially compared to a mechanism where a trigger-like mechanism is required: code can stop before the retry is triggered.
2. It offers natural throttling in the case of many emails, which is exactly a case of when errors can happen. There is little chance the servers (both ours and those of GOV.UK Notify) get overloaded by the sync task, especially if retrying with exponential backoff as is the default behaviour.
3. It forces state to be stored in Django models/the database for observability as opposed to what can be more emphemeral via the triggering of background tasks.

In terms of push-vs-pull, the happy path email is sent via a push mechansim triggering a background worker, offering the typical immediacy benefits, but retries are by the sync task "pulling" from the database, offering the typical stability benefits of that kind of architecture.

Why background tasks at all?

Emails are sent from a background task rather than from the request/response cycle for several reasons all releated to a good user experience even in the case of problems with infrastructure:

• The connection to GOV.UK Notify can theoretically be slow; its Python API is documented to have a default timeout of 30 seconds. This is beyond what is typically acceptable in a request/response cycle from a UX point of view.
• GOV.UK Notify, just like any service, can go down.
• The network infrastructure between our server and GOV.UK Notify can go down, for example during infrastructure changes.
• And even if it didn't go down, GOV.UK Notify enforces rate limits.

Why store data at all?

Observability without depending on another system helps debug issues, and being able to do this without engineer level access to low-level logs is incredible powerful. This includes users of your application because emails can be tied to standard Django models and surfaces as you see fit.

Also GOV.UK Notify does not track failures of sending emails due to network connection between your project and GOV.UK Notify, due to authentication issues, validation problems, or rate limiting. In order to track and debug such failures, they have to be stored somewhere.

License

MIT