leaf-portal

Web portal for the LEAF framework. Built with NiceGUI and asyncpg, backed by a TimescaleDB/PostgreSQL database.

Features

• Organisation & department management — hierarchical grouping of entities
• User management — superadmin, org admin, and regular users with bcrypt-hashed passwords; admin impersonation
• Access management — time-windowed grants per department/entity
• Entity management — hide entities from regular users and the sensor catalog
• Raw data retention — per-entity override of how long raw sensor_data is kept before a scheduled purge (aggregated history is always kept)
• MQTT accounts — department-scoped MQTT credentials for VerneMQ's Postgres-auth plugin, when configured (optional; hidden if the deployment doesn't have vmq_auth_acl)
• Sensor data explorer — browse and filter readings by entity, metric, and time range (multi-select), with the same aggregation levels as Plots
• Interactive plots — Plotly-based time-series visualization
• Alarm rules — threshold-based alerts (configurable per-rule check interval) with email notifications on trigger and auto-resolve
• API token management — generate and revoke tokens for REST API access
• REST API — token-authenticated endpoints for sensor data retrieval (Swagger UI at /api/docs)
• First-run setup wizard — browser-based DB connection and superadmin creation at /setup
• Password reset — email-based reset flow (/forgot-password, /reset-password)

Requirements

• Python 3.12+
• PostgreSQL 16+ or TimescaleDB
• SMTP server (optional — required for alarm emails and password reset; configured in the browser, not here — see "Outgoing mail")

Installation

From PyPI:

pip install leaf-portal

From source:

poetry install

Configuration

Create a .env file in the working directory. All variables are optional at startup — the setup wizard at /setup will prompt for DB credentials on first run and persist them to .env.

# Database (defaults shown)
PGHOST=timescaledb
PGPORT=5432
PGDATABASE=leaf
PGUSER=postgres
PGPASSWORD=

# TLS to the database. Unset falls back to libpq's "prefer", which silently
# accepts an unencrypted connection -- fine for a loopback/dev database, but
# anything weaker than verify-ca against a remote host logs a warning at
# startup. verify-full authenticates the server as well as encrypting the
# connection; point PGSSLROOTCERT at your CA bundle if it isn't a public CA.
# One of: disable | require | verify-ca | verify-full
PGSSLMODE=verify-full
PGSSLROOTCERT=/path/to/ca.crt

# Mail is NOT configured here -- see "Outgoing mail" below.

# Portal public URL (used in password-reset emails)
PORTAL_URL=http://localhost:8081

# NiceGUI session secret — change in production
STORAGE_SECRET=change-me-in-production

# Required to complete first-run setup at /setup — without it, the setup
# wizard refuses to create the initial superadmin account. Generate a
# random value and pass it to `deploy/deploy.py run` too, so it reaches
# the container.
SETUP_TOKEN=change-me-in-production

# Demo mode -- disables self-service password/2FA changes (profile page,
# forgot/reset password, and an admin resetting their own password) so a
# public demo deployment can't have its own credentials changed out from
# under it. Off by default.
DEMO_MODE=false

# Optional, demo mode only: prefill the login form with this account so a
# visitor can sign in without being told the credentials. Both must be set,
# and they are ignored unless DEMO_MODE is on. The password is visible in the
# page source -- that is the point -- so only ever use this on a deployment
# whose data is disposable.
DEMO_EMAIL=demo@example.com
DEMO_PASSWORD=

# Grafana account mirroring (optional) — whenever the portal has a user's
# plaintext password in hand (login, password reset), it pushes a matching
# account to Grafana via its admin API, so users can log into Grafana
# directly with the same credentials. Deleting a portal user deletes the
# mirrored Grafana account too.
#
# The same sync also mirrors LEAF's organisation/department structure: one
# Grafana Org per organisation, one Team per department (named
# "ORG :: CODE", using the department's short code -- e.g. "WUR :: SSB" --
# not its full display name), with membership kept in sync on every
# login/password change. Teams are looked up by a stable
# {department_id}@leaf-dept.local marker email, not by name, so renaming a
# department or its code in LEAF updates the existing team in place instead
# of orphaning its membership under a new, empty duplicate. Org membership
# is additive-only; team membership is a full sync (joined AND left, so
# leaving a department in the portal removes the matching Grafana team too).
# Datasources are not provisioned automatically -- add one per Grafana Org
# by hand.
#
# A superadmin can also generate auto-built overview dashboards -- one per
# department, plus one per entity category -- from /admin/settings
# ("Grafana Overview Dashboards" card, only shown when GRAFANA_URL is set).
# It's an on-demand admin action, not run automatically or on login: it
# scans every department system-wide, which has nothing to do with any one
# user's login event. See leaf_portal/dashboard_gen.py -- team lookup there
# reuses this same sync's convention, so it resolves to the same team
# rather than creating a lookalike duplicate.
#
# Leave GRAFANA_URL unset to disable all of the above.
#
# Grafana's admin user-management API only accepts HTTP Basic auth from a
# Grafana server-admin user (service account tokens are rejected), so this
# must be the actual Grafana admin credentials -- the same ones set via
# GF_SECURITY_ADMIN_USER / GF_SECURITY_ADMIN_PASSWORD on the Grafana side.
# Every request this integration makes carries the Grafana admin credentials,
# and the per-user sync carries that user's plaintext portal password, so a
# plain-http:// GRAFANA_URL disables the integration outright rather than
# sending those unencrypted. Set GRAFANA_ALLOW_INSECURE_HTTP=1 to opt back in
# when the connection really is confined to a network you control -- e.g. the
# bundled Grafana reached over the internal Docker network below.
GRAFANA_URL=http://grafana:3000
GRAFANA_ALLOW_INSECURE_HTTP=1
GRAFANA_ADMIN_USER=admin
GRAFANA_ADMIN_PASSWORD=

Outgoing mail

SMTP settings live in the database, entered once through Settings → Mail by a superadmin. There is deliberately no SMTP_* environment fallback: two sources meant a rotated secret could be silently overridden by a stale row, so there is one place to look and one place to change.

The group is locked — editing it asks for your password again, since it holds a working mail credential. Test email sends to your own address and reports the error inline if it fails.

Until it is filled in, superadmins see a banner on the dashboard saying mail is not set up, and alarm notifications, invites and password resets are not sent. A later failure (expired credential, unreachable host) raises a red banner instead, and every failure is recorded in the audit log as email_delivery_failed.

Running

leaf-portal
# or
python -m leaf_portal

The portal listens on 0.0.0.0:8081 by default.

On first run, navigate to http://localhost:8081 — you will be redirected to the setup wizard to configure the database connection and create the initial superadmin account.

REST API

All endpoints require a token passed via the Authorization: Bearer <token> header.

Tokens are generated from the API Tokens page (/tokens).

The API is versioned under /api/v1. Unversioned /api/... paths still work as aliases for backward compatibility but are deprecated — use /api/v1/... for new integrations.

Interactive docs: /api/docs

Development

poetry install --with dev

# Run tests
poetry run pytest tests/ -v

# Lint and format
poetry run ruff check leaf_portal/ tests/
poetry run ruff format leaf_portal/ tests/

Database

The schema lives at deploy/deploy.sql (targets TimescaleDB) and is applied automatically by the setup wizard or on startup when the DB is reachable.

Service-account credentials live separately at deploy/pre.sql — it creates the 6 login roles (leaf_portal_user, leaf_grafana_user, ...) with passwords supplied via psql -v, and must be run once, before deploy/deploy.sql, against a bare database. deploy/deploy.sql itself contains no passwords: it only grants those already-created logins access to the schema it defines, so it stays safe to reapply (e.g. via the admin "Reapply Schema" button) without ever touching credentials.

Both files are bundled into the Docker image (/app/deploy/deploy.sql, /app/deploy/pre.sql), so downstream deployments can extract them directly from whatever image tag they run instead of keeping a separately-maintained copy that can drift out of sync — see the oak stack's schema-extract service for an example.

For CI and Docker-based local development, docker/init_db.py waits for Postgres to be ready and then applies deploy/pre.sql (with placeholder passwords — CI never actually connects as these accounts) followed by deploy/deploy.sql, since deploy.sql's grants require the 6 logins to already exist.

Backups

A full pg_dump of the database doesn't make sense once sensor_data grows large — it would re-export the entire sensor history every day. Instead, docker/backup builds a small standalone image that splits the backup in two:

1. Operational tables (organisation, department, user_account, management, alarm_*, mapper_*, api_token, ...) — these are tiny, so they're fully pg_dump'd (custom format) every day to leaf_YYYY-MM-DD.dump. sensor_data and TimescaleDB's internal chunk/catalog tables are excluded.
2. sensor_data — exported per UTC day via \copy to sensor_data_YYYY-MM-DD.csv.gz. The last ROLLING_DAYS days are re-exported (overwritten) on every run to catch late-arriving readings; older files are write-once.

The image is read-only against the database (leaf_backup_user, member of the backup_readers role created by deploy/deploy.sql) and runs once per invocation — schedule it with a Kubernetes CronJob (see docker/backup/cronjob.example.yaml) or any host cron running docker run.

docker build -t leaf-backup docker/backup

docker run --rm \
  -e PGHOST=... -e PGUSER=leaf_backup_user -e PGPASSWORD=... -e PGDATABASE=leaf \
  -e ROLLING_DAYS=3 -e KEEP_DUMPS=14 \
  -v /path/to/backups:/backups \
  leaf-backup

Or, using deploy/deploy.py (builds/pushes via build-backup, runs once via backup):

python3 deploy/deploy.py build-backup   # build & push to the registry

export PGHOST=... PGUSER=leaf_backup_user PGPASSWORD=... PGDATABASE=leaf
export BACKUP_DIR=/path/to/backups      # default: ./backups
python3 deploy/deploy.py backup

Restore:

# 1. Recreate the schema (also recreates the sensor_data hypertable)
python3 deploy/deploy.py schema

# 2. Restore operational tables
pg_restore --data-only --disable-triggers -d <db> leaf_YYYY-MM-DD.dump

# 3. Re-import sensor_data for each day
zcat sensor_data_YYYY-MM-DD.csv.gz | psql -d <db> -c "\copy sensor_data FROM STDIN WITH (FORMAT csv, HEADER true)"

The backup image pins its pg_dump/pg_restore version to the TimescaleDB version this project targets (timescale/timescaledb:2.17.2-pg16) — custom-format dumps aren't readable by an older pg_restore. To back up a different Postgres major version, change the FROM tag in docker/backup/Dockerfile and rebuild. Before dumping anything, backup.sh checks that the server's major version matches its bundled pg_dump and exits with an error (without writing any files) if they've drifted apart.

Deployment

deploy/deploy.py is a helper script for building and running the portal in production. It requires no extra dependencies beyond Docker (and psql for remote schema application).

python3 deploy/deploy.py <command>

schema requires passwords for the PostgreSQL service accounts pre.sql creates — never use defaults:

export LEAF_PORTAL_PASSWORD=...   # leaf_portal_user  (portal, read+write)
export LEAF_GRAFANA_PASSWORD=...  # leaf_grafana_user (Grafana, read-only)
export LEAF_API_PASSWORD=...      # leaf_api_user     (external API access)
export LEAF_BACKUP_PASSWORD=...   # leaf_backup_user  (backup job, read-only)
export LEAF_VERNEMQ_PASSWORD=...  # leaf_vernemq_user (VerneMQ Postgres-auth, SELECT-only on vmq_auth_acl)
export LEAF_NODERED_AUTH_PASSWORD=... # leaf_nodered_auth_user (Node-RED admin login, column-limited SELECT on user_account)

Typical production flow:

# 1. Apply the schema (once, or after schema changes)
python3 deploy/deploy.py schema

# 2. Start the portal
python3 deploy/deploy.py run

Run build only when cutting a new release.

Page routes