ICHEC Django Core

ichec-django-core is the base for ICHEC's Django web apps. It gives your app secure default settings, member and organisation models, sign-in through Keycloak, and a REST API for all of these. You write the parts that are specific to your app.

The reference app is a complete app built on it, deployed with the ichec.platform Ansible collection.

What you get

• Members and organisations. Every Django user is a Member. Members belong to organisations and groups, get notifications, and can export or erase their personal data.
• Sign-in. Keycloak through OpenID Connect, Django's own login for local work, and personal API tokens for scripts.
• A REST API. Each model has an endpoint under /api/, with an OpenAPI 3.1 schema, ETags, and errors as RFC 9457 problem details.
• Access rules. Each model declares once who may do what with it. Every list, check and available_actions field comes from that declaration.
• Status changes. A model's status changes only through named transitions. Each one is its own endpoint, such as POST /api/applications/{id}/submit/.
• Files. Uploads, image thumbnails, and delivery through Django, nginx or S3, with resumable uploads through tusd.
• Forms. An opt-in app for forms defined by JSON Schema, with drafts and file answers.
• Workflows. An opt-in app for review flows held as data, so each call sets its own steps: reviews, decisions, waits and checklists, with ranking.

Try the example app

The app directory is a minimal portal built on the library. Run it locally to see what you get before you write any code. You need uv (brew install uv).

git clone https://git.ichec.ie/platform-engineering/modules/web/ichec-django-core.git
cd ichec-django-core
uv run tox exec -e dev -- python manage.py migrate
uv run tox exec -e dev -- python manage.py createsuperuser
uv run tox -e dev

Sign in at http://localhost:8000/accounts/login/ as the admin you created, then open http://localhost:8000/api/ to browse the API. The dev env in pyproject.toml holds the development settings.

Use it in your app

This guide assumes you can build a basic Django app. If not, work through the Django tutorial first.

Add the library to your dependencies:

uv add ichec-django-core

Settings

Import the library's settings, then set the few values that depend on your project layout. Your settings file then looks like app/settings.py:

from pathlib import Path

from ichec_django_core import settings
from ichec_django_core.settings import *

BASE_DIR = Path(__file__).resolve().parent.parent

ROOT_URLCONF = "app.urls"
WSGI_APPLICATION = "app.wsgi.application"
ASGI_APPLICATION = "app.asgi.application"

TEMPLATES = settings.get_templates(BASE_DIR)
DATABASES = settings.get_databases(BASE_DIR)
STATIC_ROOT = settings.get_static_root(BASE_DIR)
MEDIA_ROOT = settings.get_media_root(BASE_DIR)

Override any other setting below the import.

The library reads its settings from environment variables. Django won't start without DJANGO_SECRET_KEY and DJANGO_ALLOWED_HOSTS, or without DJANGO_SQL_PASSWORD when it uses a database server rather than SQLite. For local work, set them in a tox env, as the dev env in pyproject.toml does. In production, the deployment passes them to the container. settings.py lists every variable and its default.

Logs go to the console, which a container sends to journald. To also write a rotating log file, set DJANGO_LOG_ROOT to a directory the app can write. If it can't, the app warns and logs to the console only.

Django's own pages send a strict Content Security Policy. Set DJANGO_CSP_REPORT_ONLY=1 to try it first. settings.py shows how to extend it.

URLs

Register the library's API views on your own router, and include its other views, as in app/urls.py:

from django.urls import include, path
from rest_framework import routers

from ichec_django_core.urls import register_drf_views

router = routers.DefaultRouter()
register_drf_views(router)

urlpatterns = [
    path("api/", include(router.urls)),
    path("", include("ichec_django_core.urls")),
]

Register your app's own viewsets on the same router, so the whole API sits under one /api/ root.

Errors come back as RFC 9457 problem details, application/problem+json, with a JSON Pointer to each invalid field. Set the error handlers too, so errors on /api/ paths outside DRF's views do the same:

handler404 = "ichec_django_core.views.errors.not_found"
handler500 = "ichec_django_core.views.errors.server_error"

/health/live/ and /health/ serve liveness and readiness checks. Extend HEALTH_CHECKS to add your own.

Say who may do what

Each model declares an Access policy: the roles someone can have towards a row, then the roles that may take each action. An action the policy leaves out is refused to everyone.

from django.db import models

from ichec_django_core.access import HasPermission, Owner, Policy


class Application(models.Model):
    class Access(Policy):
        applicant = Owner("applicant")
        manager = HasPermission("change_application")

        view = applicant | manager
        change = applicant.when(status="draft")
        submit = applicant
        extra_actions = ("submit",)

A viewset built on BaseModelViewSet lists only the rows the caller may view, and refuses the actions the policy doesn't allow them. policy_for(Application).table() prints the rules in words, for review and for tests. register sets the policy of a model you don't own, such as the forms app's PopulatedForm. To hide a field from callers who may see the row, such as reviewer notes from the applicant, list it in the policy's restricted_fields with the action that shows it, and add RestrictedFieldsMixin to the serializer. access.py lists the role types.

A status changes through a django-fsm-2 @transition method. The policy says who may run it, and the transition says from which states. Add TransitionsMixin to the viewset and list the transitions in transitions, and each becomes an endpoint. run_transition is the one path for every trigger, whether it comes from the API, a scheduled task or an event. See transitions.py.

A HasPermission rule reads Django permissions, which members get through their groups. Declare each group's permissions once, in the ROLES setting:

ROLES = {
    "members": ["myapp.view_application"],
    "admins": ["myapp.view_application", "myapp.change_application"],
}

Run python manage.py sync_roles on every deploy. It gives each group exactly these permissions. A system check fails on a permission that doesn't exist, and the test fixtures grant the same permissions.

Generate your app's client

Your app's API schema is schema.yaml, and its typed Python client is generated from it. Install the codegen extra in your development tools, give your app a client package, and run:

python manage.py generate_client myapp.client

It writes schema.yaml, and generated.py and registry.py in the package. Register the registry after the library's:

from ichec_django_core.client.models import register_resources

from .generated import *  # noqa: F403
from .registry import RESOURCES

register_resources(RESOURCES)

Commit what it writes. In CI, generate_client myapp.client --check fails if any of it is out of date.

If a field of yours shares a name with one of the library's, such as id_type, but has other choices, the schema fails with an enum name clash. Name your enum in your settings:

SPECTACULAR_SETTINGS = {
    **settings.SPECTACULAR_SETTINGS,
    "ENUM_NAME_OVERRIDES": {"FacilityIdTypeEnum": "myapp.models.ID_CHOICES"},
}

Sign in with Keycloak

The example app uses Django's own login, which lets anyone register. Real deployments sign users in through Keycloak with OpenID Connect (OIDC).

Register your app as a client in the Keycloak realm, then set:

WITH_OIDC=1
OIDC_RP_CLIENT_ID=my-app
OIDC_RP_CLIENT_SECRET=...
OIDC_OP_AUTHORIZATION_ENDPOINT=https://keycloak.example.org/realms/my-realm/protocol/openid-connect/auth
OIDC_OP_TOKEN_ENDPOINT=https://keycloak.example.org/realms/my-realm/protocol/openid-connect/token
OIDC_OP_USER_ENDPOINT=https://keycloak.example.org/realms/my-realm/protocol/openid-connect/userinfo
OIDC_OP_JWKS_ENDPOINT=https://keycloak.example.org/realms/my-realm/protocol/openid-connect/certs
OIDC_OP_LOGOUT_ENDPOINT=https://keycloak.example.org/realms/my-realm/protocol/openid-connect/logout

Leave DJANGO_WITH_USER_LOGIN unset in production. The built-in login locks out repeated failed sign-ins, and Django must only be reachable through nginx. To sign in through a Keycloak on your machine, with a Test realm on port 8080, run uv run tox -e dev-oidc. mozilla-django-oidc does the OIDC work, and its docs explain each setting.

Optional features

Each of these is off until you turn it on.

• Background tasks. Emails and Keycloak syncs run inline by default. To run them on a Celery worker, install the async extra and set DJANGO_TASK_BACKEND=celery, CELERY_BROKER_URL and CELERY_RESULT_BACKEND. Then call create_celery_app() in your app's celery.py, as in app/celery.py, and run celery -A your_app worker. To run your own functions the same way, decorate them with @background_task("your_app.task_name") from ichec_django_core.tasks.

• Error tracking. Install the sentry extra and set SENTRY_DSN. Reports carry no personal data.

• File storage. Uploaded files are served by Django by default. Set DJANGO_DOWNLOAD_DELIVERY to xaccel to have nginx serve them, or to s3 to keep them in object storage. The comments in settings.py explain each mode.

• Resumable uploads. Large files can go to a tusd sidecar, which resumes them after a dropped connection. Set DJANGO_TUS_UPLOADS=1, route /api/files/tus/ to tusd, and run it with -base-path /api/files/tus/ -behind-proxy -hooks-http <app>/api/tus_hook/ -hooks-enabled-events pre-create,pre-finish. tusd must write where Django reads: -upload-dir is $MEDIA_ROOT/uploads/tus, or on S3 set -s3-object-prefix uploads/tus/. The final PATCH returns the upload, as POST /api/files/ does. Browsers must send their X-CSRFToken header to tusd. The manifest's tus_uploads tells clients whether tusd is there.

• Forms. Forms defined by JSON Schema, with drafts and file answers. Add "ichec_django_core.forms" to INSTALLED_APPS, and register_drf_views and the library's URLs include the forms API. The model that owns a form, such as an application, submits it. Register a policy for PopulatedForm to say who sees and edits answers, such as change = Owner("application.applicant").when(application__status="draft"), and one for Form to say who downloads its templates. See forms/services.py.

  Forms used to be part of the core app. If one of your models links to a form, point the link at ichec_forms in a migration that depends on ("ichec_forms", "0001_initial") and sets run_before = [("ichec_django_core", "0036_move_forms_to_ichec_forms")].

• Workflows. Each call runs its applications through a flow held as data, so a portal changes a call's review without a release. Install the workflows extra, and add "simple_history", "ichec_django_core.forms" and "ichec_django_core.workflows" to INSTALLED_APPS.

  Add WorkflowOwnerMixin to the model that owns a flow, such as a call, and WorkflowSubjectMixin to the model that goes through it, such as an application. Set owner_field on the subject to its foreign key to the owner. Serve their transitions, such as submit, close, reopen and move, with TransitionsMixin. workflows/serializers.py has the request bodies for the ones that take a date or a reason.

  A flow is a list of steps of four types: collect a form from each person in a role, decide an outcome, wait for an event or a date, and work through a checklist. Each step's role is a path from the subject to members, such as call.board_members. Coordinators edit flows at /api/flows/ and /api/steps/, and assignees work their tasks at /api/tasks/. Run python manage.py run_workflow_schedule every few minutes from cron, to close calls at their closing time.

  Keep each flow in a YAML file, and load it with python manage.py load_flow FILE. Loading it again updates the flow while no call uses it. workflows/loader.py describes the format, and tests/consumer/review_flow.yaml is an example.

  A member given a task gets a notification linking to WORKFLOW_TASK_LINK, /tasks/{id} by default. Set WORKFLOW_NOTIFY_ASSIGNEES to off to send your own. To run your own code, connect to engine.assigned, engine.entered or engine.decided, such as to record the facility the chair chose.

  Reviews follow the application, not the reviewer. Make the subject an AnonymisableMixin with "tasks" in pii_forward_fields, and erasing it erases the reviews of it. A system check fails until you do. Erasing a reviewer keeps their reviews and rankings, as the record of a decision, and only their member row stops naming them. Both find the reviews in their export. See workflows/models.py and step_types.py.

• Keycloak attributes. The app can copy values such as a member's role or avatar onto their Keycloak user, so other apps in the realm can read them. Set KEYCLOAK_ATTRIBUTE_SYNC_ENABLED=1, install the keycloak extra, and list the attributes in KEYCLOAK_ATTRIBUTE_SYNC. See sso/__init__.py.

Test your app

ichec_django_core.test has helpers for your app's tests:

• AuthAPITestCase calls your API as each test user.
• PolicyEnforcementTestCase proves your API enforces each model's policy. It calls each of your viewsets as each test user, on every row, and checks what they see and may do against the policy.
• AuthorizationSweepTestCase calls every endpoint in your schema as each role. It checks each view with an access policy against the policy, and the rest against a file listing who may call them. List only your own views without a policy. The library adds its own. For a sweep of a deployed stack, python manage.py sweep_declaration writes the whole declaration.
• The checks in test/privacy.py prove that erasing a member removes all their personal data, including from your app's models.

Test user journeys, in-process and deployed

ichec_django_core.e2e runs a user journey, written as a Scenario, in process on every merge request and against a deployed stack. List your test users in an identity map, such as e2e/identities.yml, with their groups. InProcessFixtures creates them in the test database, with the permissions ROLES gives their groups. HttpFixtures finds them in the deployment. Give both your own bundles, which create what the journey needs through the API, so the journey runs the same way in both. See e2e/fixtures.py.

For the deployed tests, import the fixtures in your e2e conftest.py, and override e2e_config:

from pathlib import Path

import pytest

from ichec_django_core.e2e.pytest_plugin import *  # noqa: F403
from ichec_django_core.e2e.pytest_plugin import E2EConfig


@pytest.fixture(scope="session")
def e2e_config() -> E2EConfig:
    return E2EConfig(identity_map=Path(__file__).parent / "identities.yml")

ichec_django_core.e2e.suites has the tests every deployment runs: sign-in and the session cookie, how the SPA is served, and the authorization sweep. Subclass each in a test module. With users="run", each run creates its test users in Keycloak and deletes them after. To create a permanent user, such as a scanner's, run python -m ichec_django_core.e2e.realm apply --map FILE. e2e/pytest_plugin.py lists the variables a deployment sets.

Work on this library

Install the development tools and run the checks that CI runs:

uv sync --group dev --extra async
uv run pytest
uv run ruff format --check src tests
uv run ruff check src tests
uv run mypy src

If you use direnv, run direnv allow once and the environment activates when you enter the directory.

If you change a model or serializer, regenerate the API schema and client, and commit them:

uv run python manage.py generate_client ichec_django_core.client

Licence

Copyright of the Irish Centre for High End Computing (ICHEC), released under the MIT License. See LICENSE.