django-govuk-notify-outbox 1.5.2

A reusable Django app to send emails via GOV.UK notify robustly and with good observability

django-govuk-notify-outbox

A reusable Django app to send emails via GOV.UK notify robustly and with good observability.

---

Contents

• Features
• Prerequisites
• Installation
• Usage
  • Fire and forget
  • Attaching to your own models
• Email status descriptions
  • Happy path
  • Happy and unhappy paths
• Configuration
• Architecture
  • Why the sync task?
  • Why background tasks at all?
  • Why store data at all?
• License

---

Features

• Sends emails outside of the request/response cycle (via Celery)
• Automatically retries communication with GOV.UK Notify up to a maximum number / maximum amount of time
• Usage is consistent with the send_email_notification function from notifications-python-client
• Emails and updates from GOV.UK Notify (including HTTP errors and Python exceptions) are stored in Django models...
• ... so can be linked (via foreign key or many-to-many relation) to other models to be used in the user-facing application
• ... and are visible in the Django admin interface

Prerequisites

• An existing (even if newly created) Django project
• ... which is configured with Celery and Celery Beat to run background tasks
• A GOV.UK Notify service API Key

Installation

1. Install django-govuk-notify-outbox from PyPI.

   Typically you would do this by adding django-govuk-notify-outbox as a dependency to your existing Django project, for example with uv:

   uv add django-govuk-notify-outbox
   

2. Add django_govuk_notify_outbox to settings.INSTALLED_APPS

   INSTALLED_APPS = [
       # ...
       "django_govuk_notify_outbox",
   ]
   

3. Add settings.DJANGO_GOVUK_NOTIFY_OUTBOX__API_KEY with your GOV.UK Notify API key, typically from an environment variable.

   import os
   
   DJANGO_GOVUK_NOTIFY_OUTBOX__API_KEY = os.environ.get('DJANGO_GOVUK_NOTIFY_OUTBOX__API_KEY')
   

4. Add django_govuk_notify_outbox.tasks.sync to settings.CELERY_BEAT_SCHEDULE on a frequent schedule, for example every 5 seconds. If there is nothing to do, this will finish quickly and without connecting to GOV.UK notify.

   CELERY_BEAT_SCHEDULE = {
       "sync": {
           "task": "django_govuk_notify_outbox.tasks.sync",
           "schedule": 5,
       },
   }
   

5. Run python manage.py migrate to create the models. Often projects are setup to do this automatically on startup.

Usage

Fire and forget

Once installed, emails can be sent from any part of your Django project with the django_govuk_notify_outbox.utils.send_email_notification function, which is designed to be consistent with the "fire and forget"-style send_email_notification function from notifications-python-client

from django_govuk_notify_outbox.utils import send_email_notification

send_email_notification(
    email_address="[e-mail]",
    template_id="9d751e0e-f929-4891-82a1-a3e1c3c18ee3",
)

Attaching to your own models

The return value of send_email_notification is an instance of the Django model django_govuk_notify_outbox.models.EmailMessage, and it can be attached to your own models in the usual way via ForeignKey or ManyToManyField. Each EmailMessage is updated through the email sending process until is is delivered or failed.

For example:

# In your app's models.py
from django.db import models
from django_govuk_notify_outbox.models import EmailMessage

class MyModel(models.Model):
    email_messages = models.ManyToManyField(EmailMessage)

# Anywhere in your app
from django_govuk_notify_outbox.utils import send_email_notification

email_message = send_email_notification(
    email_address="[e-mail]",
    template_id="9d751e0e-f929-4891-82a1-a3e1c3c18ee3",
)
my_model_instance = MyModel.create()
my_model_instance.email_messages.add(email_message)
print(email_message.status)  # See below

Email status descriptions

Each EmailMessage.status field can be equal to one of 10 possible values. For the value that reflect a GOV.UK Notify status, see https://docs.notifications.service.gov.uk/python.html#email-status-descriptions for details.

State	Description
sending-to-notify	The email is about to be sent to GOV.UK Notify (or has very recently been sent).
retrying-send-to-notify	The email has failed sending to GOV.UK Notify at least once and will be retried (or has very recently been retried).
failed-send-to-notify	The email has failed sending to GOV.UK Notify and will not be retried.
sent-to-notify	The email has been sent to GOV.UK Notify
notify-created	GOV.UK notify reports the email in the created state, which essentially means queued: send will be attempted in the next few seconds.
notify-sending	GOV.UK notify reports the email in the sending state.
notify-delivered	GOV.UK notify reports the email in the delivered state.
notify-temporary-failure	GOV.UK notify reports the email in the temporary-failure state. Note that this does not mean that the email will be retried automatically, but more reflects that if another identical email is sent, it may succeed.
notify-technical-failure	GOV.UK Notify reports the email in the technical-failure state, which indicates a technical issue between Notify and its email provider. It will not be retried automatically.
notify-permanent-failure	GOV.UK notify reports the email in the permanent-failure state, which indicates it is not likely to succeed if another identical email is sent.

Happy path

5 statuses make up the happy path when there are no problems sending an email:

stateDiagram-v2

    sendingToNotify: sending-to-notify
    sentToNotify: sent-to-notify
    notifyCreated: notify-created
    notifySending: notify-sending
    notifyDelivered: notify-delivered

    [*] --> sendingToNotify
    sendingToNotify --> sentToNotify
    sentToNotify --> notifyCreated
    notifyCreated --> notifySending
    notifySending --> notifyDelivered
    notifyDelivered --> [*]

Happy and unhappy paths

How EmailMessage.status can change in all cases of both success and failure is shown in the following diagram:

stateDiagram-v2

    sendingToNotify: sending-to-notify
    retryingSendToNotify: retrying-send-to-notify
    sentToNotify: sent-to-notify
    failedSendToNotify: failed-send-to-notify
    notifyCreated: notify-created
    notifySending: notify-sending
    notifyDelivered: notify-delivered
    notifyPermanentFailure: notify-permanent-failure
    notifyTemporaryFailure: notify-temporary-failure
    notifyTechnicalFailure: notify-technical-failure

    [*] --> sendingToNotify

    sendingToNotify --> retryingSendToNotify
    retryingSendToNotify --> failedSendToNotify
    sendingToNotify --> sentToNotify
    retryingSendToNotify --> sentToNotify

    failedSendToNotify --> [*]

    sentToNotify --> notifyCreated
    notifyCreated --> notifySending

    notifySending --> notifyDelivered
    notifySending --> notifyPermanentFailure
    notifySending --> notifyTemporaryFailure
    notifySending --> notifyTechnicalFailure

    notifyPermanentFailure --> [*]
    notifyTemporaryFailure --> [*]
    notifyTechnicalFailure --> [*]

    notifyDelivered --> [*]

Configuration

Configuration is via settings that go into your Django project's settings.py

Setting	Description
DJANGO_GOVUK_NOTIFY_OUTBOX__API_KEY	The GOV.UK Notify API key for the service
DJANGO_GOVUK_NOTIFY_OUTBOX__SEND_TO_NOTIFY_RETRY_INTERVALS	A list of how long to wait in seconds between retrying send to GOV.UK Notify Default: [5,10,60,120,240,480,960,1920]
DJANGO_GOVUK_NOTIFY_OUTBOX__SEND_TO_NOTIFY_ABANDON_AFTER	How long in seconds to wait to abandon send to GOV.UK notify, even if there are remaining retries according to the DJANGO_GOVUK_NOTIFY_OUTBOX__SEND_TO_NOTIFY_RETRY_INTERVALS setting. This is used to not send emails if the background worker has been offline for a long period of time and so the emails are no longer relevant. Default: 3600

Architecture

Why the sync task?

The sync task does two things. It retries sending to GOV.UK Notify, and it fetches updates on sent emails. There are are three reasons for this:

1. It is the most robust way to make sure retries are attempted, especially compared to a mechanism where a trigger-like mechanism is required: code can stop before the retry is triggered.
2. It offers natural throttling in the case of many emails, which is exactly a case of when errors can happen. There is little chance the servers (both ours and those of GOV.UK Notify) get overloaded by the sync task, especially if retrying with exponential backoff as is the default behaviour.
3. It forces state to be stored in Django models/the database for observability as opposed to what can be more emphemeral via the triggering of background tasks.

In terms of push-vs-pull, the happy path email is sent via a push mechansim triggering a background worker, offering the typical immediacy benefits, but retries are by the sync task "pulling" from the database, offering the typical stability benefits of that kind of architecture.

Why background tasks at all?

Emails are sent from a background task rather than from the request/response cycle for several reasons all releated to a good user experience even in the case of problems with infrastructure:

• The connection to GOV.UK Notify can theoretically be slow; its Python API is documented to have a default timeout of 30 seconds. This is beyond what is typically acceptable in a request/response cycle from a UX point of view.
• GOV.UK Notify, just like any service, can go down.
• The network infrastructure between our server and GOV.UK Notify can go down, for example during infrastructure changes.
• And even if it didn't go down, GOV.UK Notify enforces rate limits.

Why store data at all?

Observability without depending on another system helps debug issues, and being able to do this without engineer level access to low-level logs is incredible powerful. This includes users of your application because emails can be tied to standard Django models and surfaces as you see fit.

Also GOV.UK Notify does not track failures of sending emails due to network connection between your project and GOV.UK Notify, due to authentication issues, validation problems, or rate limiting. In order to track and debug such failures, they have to be stored somewhere.

License

MIT