ICHEC Django Core

ichec-django-core is the base for ICHEC's Django web apps. It gives your app secure default settings, member and organisation models, sign-in through Keycloak, and a REST API for all of these. You write the parts that are specific to your app.

The reference app is a complete app built on it, deployed with the ichec.platform Ansible collection.

What you get

Some details the diagram doesn't show:

• Every Django user is a Member, who can export or erase their personal data.
• Each model declares once who may do what with it. Every list, check and available_actions field comes from that declaration.
• A model's status changes only through named transitions. Each one is its own endpoint, such as POST /api/applications/{id}/submit/.
• Files are delivered through Django, nginx or S3, and large uploads resume through tusd.
• Forms and workflows are opt-in apps.

Try the example app

The app directory is a minimal portal built on the library. Run it locally to see what you get before you write any code. You need uv (brew install uv).

git clone https://git.ichec.ie/platform-engineering/modules/web/ichec-django-core.git
cd ichec-django-core
uv run tox exec -e dev -- python manage.py migrate
uv run tox exec -e dev -- python manage.py createsuperuser
uv run tox -e dev

Sign in at http://localhost:8000/accounts/login/ as the admin you created, then open http://localhost:8000/api/ to browse the API. The dev env in pyproject.toml holds the development settings.

Use it in your app

This guide assumes you can build a basic Django app. If not, work through the Django tutorial first.

Add the library to your dependencies:

uv add ichec-django-core

Settings

Import the library's settings, then set the few values that depend on your project layout. Your settings file then looks like app/settings.py:

from pathlib import Path

from ichec_django_core import settings
from ichec_django_core.settings import *

BASE_DIR = Path(__file__).resolve().parent.parent

ROOT_URLCONF = "app.urls"
WSGI_APPLICATION = "app.wsgi.application"
ASGI_APPLICATION = "app.asgi.application"

TEMPLATES = settings.get_templates(BASE_DIR)
DATABASES = settings.get_databases(BASE_DIR)
STATIC_ROOT = settings.get_static_root(BASE_DIR)
MEDIA_ROOT = settings.get_media_root(BASE_DIR)

Override any other setting below the import.

The library reads its settings from environment variables. Django won't start without DJANGO_SECRET_KEY and DJANGO_ALLOWED_HOSTS, or without DJANGO_SQL_PASSWORD when it uses a database server rather than SQLite. For local work, set them in a tox env, as the dev env in pyproject.toml does. In production, the deployment passes them to the container. settings.py lists every variable and its default.

The environment holds deployment settings only. What an admin changes, the portal's name, contact address and help link, is in the database, at /api/portal/, for holders of change_portal. The manifest sends it to the frontend. Members with the receive_signup_alerts permission get an email when someone new signs in.

Email and links

Set two variables in production. A deploy system check fails without them:

• DJANGO_PORTAL_URL, the portal's public URL, such as https://portal.example.org. Every link the app sends people is built from it with portal_url(path) from ichec_django_core.services.links. Use that helper rather than a setting of your own.
• DJANGO_DEFAULT_FROM_EMAIL, the address emails come from. Emails show the portal's name with it, unless the value names itself, as Team <team@example.org> does.

DJANGO_PORTAL_LOGO_URL adds a logo to emails.

Send every email with send_email(kind, to, context) from ichec_django_core.services.email. It renders emails/<kind>.subject.txt, emails/<kind>.txt and emails/<kind>.html, which extend emails/base.txt and emails/base.html. To add a kind, put its three templates in your app's templates/emails/. In them, {% load ichec_email %} gives the portal_url filter for links, such as {{ link|portal_url }}. Text parts aren't HTML-escaped. The email goes after the transaction commits, through the task backend, and a failure is logged rather than raised, so it never fails a request or a sign-in.

Logs go to the console, which a container sends to journald. To also write a rotating log file, set DJANGO_LOG_ROOT to a directory the app can write. If it can't, the app warns and logs to the console only.

Django's own pages send a strict Content Security Policy. Set DJANGO_CSP_REPORT_ONLY=1 to try it first. settings.py shows how to extend it.

URLs

Register the library's API views on your own router, and include its other views, as in app/urls.py:

from django.urls import include, path
from rest_framework import routers

from ichec_django_core.urls import register_drf_views

router = routers.DefaultRouter()
register_drf_views(router)

urlpatterns = [
    path("api/", include(router.urls)),
    path("", include("ichec_django_core.urls")),
]

Register your app's own viewsets on the same router, so the whole API sits under one /api/ root.

Errors come back as RFC 9457 problem details, application/problem+json, with a JSON Pointer to each invalid field. Set the error handlers too, so errors on /api/ paths outside DRF's views do the same:

handler404 = "ichec_django_core.views.errors.not_found"
handler500 = "ichec_django_core.views.errors.server_error"

/health/live/ and /health/ serve liveness and readiness checks. Extend HEALTH_CHECKS to add your own.

If Django serves your Angular app's index.html, add the library's route last, and set FRONTEND_INDEX to the file:

from ichec_django_core.views.spa import spa_route

urlpatterns += [spa_route("reports/")]

It serves the SPA for every path except the library's and the ones you pass. It sends none of Django's Content Security Policy, which would block the inline styles Angular adds, so nginx must set the SPA's policy. Don't write your own view for this: one that sends Django's policy serves the app unstyled.

Say who may do what

Each model declares an Access policy: the roles someone can have towards a row, then the roles that may take each action. An action the policy leaves out is refused to everyone.

from django.db import models

from ichec_django_core.access import HasPermission, Owner, Policy


class Application(models.Model):
    class Access(Policy):
        applicant = Owner("applicant")
        manager = HasPermission("change_application")

        view = applicant | manager
        change = applicant.when(status="draft")
        submit = applicant
        extra_actions = ("submit",)

A viewset built on BaseModelViewSet lists only the rows the caller may view, and refuses the actions the policy doesn't allow them. policy_for(Application).table() prints the rules in words, for review and for tests. register sets the policy of a model you don't own, such as the forms app's PopulatedForm. To hide a field from callers who may see the row, such as reviewer notes from the applicant, list it in the policy's restricted_fields with the action that shows it, and add RestrictedFieldsMixin to the serializer. With Allowed("application"), a row follows a related row's policy, such as a form its application's. access.py lists the role types.

A status changes through a django-fsm-2 @transition method. The policy says who may run it, and the transition says from which states. Add TransitionsMixin to the viewset and list the transitions in transitions, and each becomes an endpoint. run_transition is the one path for every trigger, whether it comes from the API, a scheduled task or an event. See transitions.py.

A body field named reason goes in the row's history, if the model keeps one with django-simple-history. To act on a transition, such as to tell someone, connect to transition_saved. It sends the saved instance, the transition's name, the reason, or "" without one, and the user who ran it, or None when the system did:

from django.dispatch import receiver
from ichec_django_core.transitions import transition_saved
from ichec_django_core.workflows.models import Task


@receiver(transition_saved, sender=Task)
def tell_the_coordinator(sender, instance, name, reason, user, **kwargs):
    if name == "decline":
        ...  # Say who declined, and the reason they gave.

A HasPermission rule reads Django permissions, which members get through their groups. Declare each group's permissions once, in the ROLES setting:

ROLES = {
    "members": ["myapp.view_application"],
    "admins": ["myapp.view_application", "myapp.change_application"],
}

Run python manage.py sync_roles on every deploy. It gives each group exactly these permissions. A system check fails on a permission that doesn't exist, and the test fixtures grant the same permissions.

Who sees a member

Other objects name people with a member reference, which reads as {url, id, name} and takes the member's URL on write. Use MemberReferenceField for your own member fields. The reference shows wherever its object does, so a reviewer sees an applicant's name without seeing their member record.

By default a member record is seen only by the member and by holders of view_member. Phone, identifiers and image need view_account, which the member and holders of view_member_account or change_member have. Groups also show to holders of change_member_groups. To let people see the members they work with, register your own policy. Subclass the library's, so it keeps the account rules:

from ichec_django_core.access import Allowed, HasPermission, register
from ichec_django_core.models import Member


class Members(Member.Access):
    # Anyone who may see an application sees its applicant.
    view = (
        Member.Access.themself
        | HasPermission("view_member")
        | Allowed("application")
    )


register(Member, Members)

Allowed follows a reverse relation by its query name, here the reverse of Application.applicant, so a member shows to whoever may view one of their applications.

Give members roles

A member's groups give them their roles. Holders of change_member_groups set them with PUT /api/members/{id}/groups/, and an invitation to /api/invites/ takes groups too. Either way, a caller may add or remove only the groups whose permissions they hold themselves, so nobody grants more than they have. Changing a group's user_set at /api/groups/{id}/ follows the same rule. change_member_groups doesn't let them change a member's name or other details, which needs change_member. Give it to your admin group in ROLES.

Generate your app's client

Your app's API schema is schema.yaml, and its typed Python client is generated from it. Install the codegen extra in your development tools, give your app a client package, and run:

python manage.py generate_client myapp.client

It writes schema.yaml, and generated.py and registry.py in the package. Register the registry after the library's:

from ichec_django_core.client.models import register_resources

from .generated import *  # noqa: F403
from .registry import RESOURCES

register_resources(RESOURCES)

Commit what it writes. In CI, generate_client myapp.client --check fails if any of it is out of date.

If a field of yours shares a name with one of the library's, such as id_type, but has other choices, the schema fails with an enum name clash. Name your enum in your settings:

SPECTACULAR_SETTINGS = {
    **settings.SPECTACULAR_SETTINGS,
    "ENUM_NAME_OVERRIDES": {"FacilityIdTypeEnum": "myapp.models.ID_CHOICES"},
}

Sign in with Keycloak

The example app uses Django's own login, which lets anyone register. Real deployments sign users in through Keycloak with OpenID Connect (OIDC).

Register your app as a client in the Keycloak realm, then set:

WITH_OIDC=1
OIDC_RP_CLIENT_ID=my-app
OIDC_RP_CLIENT_SECRET=...
OIDC_OP_AUTHORIZATION_ENDPOINT=https://keycloak.example.org/realms/my-realm/protocol/openid-connect/auth
OIDC_OP_TOKEN_ENDPOINT=https://keycloak.example.org/realms/my-realm/protocol/openid-connect/token
OIDC_OP_USER_ENDPOINT=https://keycloak.example.org/realms/my-realm/protocol/openid-connect/userinfo
OIDC_OP_JWKS_ENDPOINT=https://keycloak.example.org/realms/my-realm/protocol/openid-connect/certs
OIDC_OP_LOGOUT_ENDPOINT=https://keycloak.example.org/realms/my-realm/protocol/openid-connect/logout

Leave DJANGO_WITH_USER_LOGIN unset in production. The built-in login locks out repeated failed sign-ins, and Django must only be reachable through nginx. To sign in through a Keycloak on your machine, with a Test realm on port 8080, run uv run tox -e dev-oidc. mozilla-django-oidc does the OIDC work, and its docs explain each setting.

Optional features

Each of these is off until you turn it on.

• Background tasks. Emails and Keycloak syncs run inline by default. To run them on a Celery worker, install the async extra and set DJANGO_TASK_BACKEND=celery, CELERY_BROKER_URL and CELERY_RESULT_BACKEND. Then call create_celery_app() in your app's celery.py, as in app/celery.py, and run celery -A your_app worker. To run your own functions the same way, decorate them with @background_task("your_app.task_name") from ichec_django_core.tasks.

• Facet counts. A SearchableModelViewSet counts the values of the filters it names in facet_fields, such as {"country": Facet("address__country")}. Asked with ?facets=country, the page gains facets beside results. Each count applies the search and the other filters, but not its own. views/facets.py explains how values are named.

• Error tracking. Install the sentry extra and set SENTRY_DSN. Reports carry no personal data.

• File storage. Every upload is checked by its content, not just its extension: images must open, PDFs must parse without a password, and Word, Excel and OpenDocument files must hold their type's parts. A file answer must also be of a type its field's x-file.accept lists. Uploaded files are served by Django by default. Set DJANGO_DOWNLOAD_DELIVERY to xaccel to have nginx serve them, or to s3 to keep them in object storage. The comments in settings.py explain each mode.

• Resumable uploads. Large files can go to a tusd sidecar, which resumes them after a dropped connection. Set DJANGO_TUS_UPLOADS=1, route /api/files/tus/ to tusd, and run it with -base-path /api/files/tus/ -behind-proxy -hooks-http <app>/api/tus_hook/ -hooks-enabled-events pre-create,pre-finish. tusd must write where Django reads: -upload-dir is $MEDIA_ROOT/uploads/tus, or on S3 set -s3-object-prefix uploads/tus/. The final PATCH returns the upload, as POST /api/files/ does. Browsers must send their X-CSRFToken header to tusd. The manifest's tus_uploads tells clients whether tusd is there, and its max_upload_bytes gives the largest file the server takes (DJANGO_MAX_UPLOAD_BYTES, 50 MiB by default).

• Error reporting. Set DJANGO_ERROR_REPORTING_DSN to a Sentry DSN, and optionally DJANGO_ERROR_REPORTING_ENVIRONMENT, and the manifest's error_reporting tells the browser app where to report its errors. Unset, it's null, and the app reports nothing.

• Forms. Forms defined by JSON Schema, with drafts and file answers. Add "ichec_django_core.forms" to INSTALLED_APPS, and register_drf_views and the library's URLs include the forms API. The model that owns a form, such as an application, submits it. Register a policy for PopulatedForm to say who sees and edits answers, such as change = Owner("application.applicant").when(application__status="draft"), and one for Form to say who downloads its templates. See forms/services.py.

  To show answers outside the form, such as in a PDF or an email, take readable_answers(form, keep_empty=True) and render them with forms/render.py, or in a template with {% load ichec_forms %} and {{ answers|answers_html }}. It keeps sections, line breaks and cleaned rich text, and shows "Not answered". Don't print Answer.value yourself: rich text holds HTML. Dates follow LANGUAGE_CODE, which is Irish English, so they read day first.

  Forms used to be part of the core app. If one of your models links to a form, point the link at ichec_forms in a migration that depends on ("ichec_forms", "0001_initial") and sets run_before = [("ichec_django_core", "0036_move_forms_to_ichec_forms")].

• Workflows. Each call runs its applications through a flow held as data, so a portal changes a call's review without a release. Install the workflows extra, and add "simple_history", "ichec_django_core.forms" and "ichec_django_core.workflows" to INSTALLED_APPS.

  Add WorkflowOwnerMixin to the model that owns a flow, such as a call, and WorkflowSubjectMixin to the model that goes through it, such as an application. Set owner_field on the subject to its foreign key to the owner. Serve their transitions, such as submit, close, reopen and move, with TransitionsMixin. workflows/serializers.py has the request bodies for the ones that take a date or a reason. Serve withdraw and proceed with ReasonSerializer, so people can say why. A task's decline takes an optional reason.

  A flow is a list of steps of four types: collect a form from each person in a role, decide an outcome, wait for an event or a date, and work through a checklist. Each step's role is a path from the subject to members, such as call.board_members. Coordinators edit flows at /api/flows/ and /api/steps/, and assignees work their tasks at /api/tasks/. Run python manage.py run_workflow_schedule every few minutes from cron, to close calls at their closing time.

  A call's copy of a flow has its own step forms, which /api/forms/ changes. GET /api/forms/ lists the forms the Form policy lets the caller see, for choosing a step's form, with title, flow, template and step_type filters. A step given a form it doesn't have stores its own copy, so flows never share one. Whoever may change a call may change its flow and steps until it opens, by the call's own policy.

  Keep each flow in a YAML file, and list it in WORKFLOW_TEMPLATES, which loads it after each migrate, or load it with python manage.py load_flow FILE. A loaded flow is a template. Each call gets its own copy when it is created, so a coordinator can change one call's steps and leave the others alone. Loading the file again updates the template, not the copies. workflows/loader.py describes the format, and tests/consumer/review_flow.yaml is an example.

  Some things need a person's judgement, so the flow leaves them to someone the policy allows:

  • A collect step with declines: wait holds when someone declines, until a coordinator assigns a replacement with replaces, or someone runs proceed on the subject to move on without one.
  • proceed finishes any step but a decision early. Late responses still count, except in a step that ends at an event, such as the close.
  • reopen gives one subject more time after its call closes, whether it is still a draft or already in review.
  • move corrects a subject's place, including after a withdrawal.

  A member given a task gets a notification linking to WORKFLOW_TASK_LINK, /tasks/{id} by default. It names the subject by workflow_title() and the step by its label, says what to do and when it is due. Tasks one person gets in one step at once, such as at a call's close, come in one email. Set a step's notify to false to send your own for that step. To run your own code, connect to engine.assigned, engine.entered or engine.decided, such as to record the facility the chair chose.

  run_workflow_schedule also sends reminders, WORKFLOW_REMINDER_DAYS before a task is due and before a draft's call closes. A draft's reminder goes to the member in submitter_field, or pii_subject_field if unset. Override workflow_link() on the subject to give it the subject's portal page, and workflow_title() on the owner to name the call.

  A task is due when its step ends, or by its due setting, such as due: {after: request_end_date, days: 30} for 30 days after a date on the subject. A step done with status: completed, such as done: {responses: one, status: completed}, completes an accepted subject. The new completed status changes the subject's status choices, so run makemigrations for the subject model.

  Send your own notifications with notify(member, kind, subject, title=, message=, link=, about=, **facts) from ichec_django_core.services.notifications. Its email uses your templates/emails/notifications/<kind>.*, or the generic ones. Name records by title, never by id. A title is often personal data, so it goes in title, subject or message, which erasure blanks, and never in the facts. Pass the record as about, and list "notifications" in the subject's pii_forward_fields, so erasing the subject erases the notifications that name it. A task's notifications already follow its subject.

  The subject's policy says who reads the work done on it, with a view_tasks action, such as the chair of the call, and who assigns and cancels its tasks, with an assign_tasks action, such as the call's coordinator. Holders of add_task and change_task still may. A task lists assign in its available_actions for those who may assign a replacement.

  A step that needs responses and has nobody left to give them, such as one waiting for a replacement or for a coordinator to assign, is blocked. Add WorkflowProgressMixin to the subject's serializer and "workflow_progress" to its fields to show the step's counts, with declined and unassigned tasks, and blocked. Register a policy for PopulatedForm with view = Allowed("task") and change = Allowed("task", "change"), so each task's form follows its task.

  Reviews follow the application, not the reviewer. Make the subject an AnonymisableMixin with "tasks" in pii_forward_fields, and erasing it erases the reviews of it. A system check fails until you do. Erasing a reviewer keeps their reviews and rankings, as the record of a decision, and only their member row stops naming them. Both find the reviews in their export. See workflows/models.py and step_types.py.

• Keycloak attributes. The app can copy values such as a member's role or avatar onto their Keycloak user, so other apps in the realm can read them. Set KEYCLOAK_ATTRIBUTE_SYNC_ENABLED=1, install the keycloak extra, and list the attributes in KEYCLOAK_ATTRIBUTE_SYNC. See sso/__init__.py.

Test your app

ichec_django_core.test has helpers for your app's tests:

• AuthAPITestCase calls your API as each test user.
• PolicyEnforcementTestCase proves your API enforces each model's policy. It calls each of your viewsets as each test user, on every row, and checks what they see and may do against the policy.
• AuthorizationSweepTestCase calls every endpoint in your schema as each role. It checks each view with an access policy against the policy, and the rest against a file listing who may call them. List only your own views without a policy. The library adds its own. For a sweep of a deployed stack, python manage.py sweep_declaration writes the whole declaration.
• The checks in test/privacy.py prove that erasing a member removes all their personal data, including from your app's models.

Test user journeys, in-process and deployed

ichec_django_core.e2e runs a user journey, written as a Scenario, in process on every merge request and against a deployed stack. List your test users in an identity map, such as e2e/identities.yml, with their groups. InProcessFixtures creates them in the test database, with the permissions ROLES gives their groups. HttpFixtures finds them in the deployment. Give both your own bundles, which create what the journey needs through the API, so the journey runs the same way in both. See e2e/fixtures.py.

For the deployed tests, import the fixtures in your e2e conftest.py, and override e2e_config:

from pathlib import Path

import pytest

from ichec_django_core.e2e.pytest_plugin import *  # noqa: F403
from ichec_django_core.e2e.pytest_plugin import E2EConfig


@pytest.fixture(scope="session")
def e2e_config() -> E2EConfig:
    return E2EConfig(identity_map=Path(__file__).parent / "identities.yml")

ichec_django_core.e2e.suites has the tests every deployment runs: sign-in and the session cookie, how the SPA is served and that no Content Security Policy blocks it, and the authorization sweep. Subclass each in a test module. With users="run", each run creates its test users in Keycloak and deletes them after. To create a permanent user, such as a scanner's, run python -m ichec_django_core.e2e.realm apply --map FILE. e2e/pytest_plugin.py lists the variables a deployment sets.

Work on this library

Install the development tools and run the checks that CI runs:

uv sync --group dev --extra async
uv run pytest
uv run ruff format --check src tests
uv run ruff check src tests
uv run mypy src

If you use direnv, run direnv allow once and the environment activates when you enter the directory.

If you change a model or serializer, regenerate the API schema and client, and commit them:

uv run python manage.py generate_client ichec_django_core.client

Licence

Copyright of the Irish Centre for High End Computing (ICHEC), released under the MIT License. See LICENSE.