leaf-portal

Web portal for the LEAF framework. Built with NiceGUI and asyncpg, backed by a TimescaleDB/PostgreSQL database.

Features

• Organisation & department management — hierarchical grouping of entities
• User management — superadmin, org admin, and regular users with bcrypt-hashed passwords; admin impersonation
• Access management — time-windowed grants per department/entity
• Entity management — hide entities from regular users and the sensor catalog
• Raw data retention — per-entity override of how long raw sensor_data is kept before a scheduled purge (aggregated history is always kept)
• MQTT accounts — department-scoped MQTT credentials for VerneMQ's Postgres-auth plugin, when configured (optional; hidden if the deployment doesn't have vmq_auth_acl)
• Sensor data explorer — browse and filter readings by entity, metric, and time range (multi-select), with the same aggregation levels as Plots
• Interactive plots — Plotly-based time-series visualization
• Alarm rules — threshold-based alerts (configurable per-rule check interval) with email notifications on trigger and auto-resolve
• API token management — generate and revoke tokens for REST API access
• REST API — token-authenticated endpoints for sensor data retrieval (Swagger UI at /api/docs)
• First-run setup wizard — browser-based DB connection and superadmin creation at /setup
• Password reset — email-based reset flow (/forgot-password, /reset-password)

Requirements

• Python 3.12+
• PostgreSQL 16+ or TimescaleDB
• SMTP server (optional — required for alarm emails and password reset)

Installation

From PyPI:

pip install leaf-portal

From source:

poetry install

Configuration

Create a .env file in the working directory. All variables are optional at startup — the setup wizard at /setup will prompt for DB credentials on first run and persist them to .env.

# Database (defaults shown)
PGHOST=timescaledb
PGPORT=5432
PGDATABASE=leaf
PGUSER=postgres
PGPASSWORD=

# Mail (required for alarm emails and password reset)
SMTP_HOST=smtp.example.com
SMTP_PORT=587
SMTP_USER=user@example.com
SMTP_PASSWORD=secret
SMTP_FROM=noreply@example.com

# Portal public URL (used in password-reset emails)
PORTAL_URL=http://localhost:8081

# NiceGUI session secret — change in production
STORAGE_SECRET=change-me-in-production

# Grafana account mirroring (optional) — whenever the portal has a user's
# plaintext password in hand (login, password reset), it pushes a matching
# account to Grafana via its admin API, so users can log into Grafana
# directly with the same credentials. Deleting a portal user deletes the
# mirrored Grafana account too.
#
# The same sync also mirrors LEAF's organisation/department structure: one
# Grafana Org per organisation, one Team per department (named "ORG :: DEPT"),
# with membership kept in sync on every login/password change. Org membership
# is additive-only; team membership is a full sync (joined AND left, so
# leaving a department in the portal removes the matching Grafana team too).
# Datasources are not provisioned automatically -- add one per Grafana Org
# by hand.
#
# Leave GRAFANA_URL unset to disable all of the above.
#
# Grafana's admin user-management API only accepts HTTP Basic auth from a
# Grafana server-admin user (service account tokens are rejected), so this
# must be the actual Grafana admin credentials -- the same ones set via
# GF_SECURITY_ADMIN_USER / GF_SECURITY_ADMIN_PASSWORD on the Grafana side.
GRAFANA_URL=http://grafana:3000
GRAFANA_ADMIN_USER=admin
GRAFANA_ADMIN_PASSWORD=

Running

leaf-portal
# or
python -m leaf_portal

The portal listens on 0.0.0.0:8081 by default.

On first run, navigate to http://localhost:8081 — you will be redirected to the setup wizard to configure the database connection and create the initial superadmin account.

REST API

All endpoints require a token passed via the Authorization: Bearer <token> header.

Tokens are generated from the API Tokens page (/tokens).

The API is versioned under /api/v1. Unversioned /api/... paths still work as aliases for backward compatibility but are deprecated — use /api/v1/... for new integrations.

Interactive docs: /api/docs

Development

poetry install --with dev

# Run tests
poetry run pytest tests/ -v

# Lint and format
poetry run ruff check leaf_portal/ tests/
poetry run ruff format leaf_portal/ tests/

Database

The schema lives at deploy/deploy.sql (targets TimescaleDB) and is applied automatically by the setup wizard or on startup when the DB is reachable.

deploy/deploy.sql is also bundled into the Docker image (/app/deploy/deploy.sql), so downstream deployments can extract it directly from whatever image tag they run instead of keeping a separately-maintained copy that can drift out of sync — see the oak stack's schema-extract service for an example.

For CI and Docker-based local development, docker/init_db.py waits for Postgres to be ready and then applies deploy/deploy.sql.

Backups

A full pg_dump of the database doesn't make sense once sensor_data grows large — it would re-export the entire sensor history every day. Instead, docker/backup builds a small standalone image that splits the backup in two:

1. Operational tables (organisation, department, user_account, management, alarm_*, mapper_*, api_token, ...) — these are tiny, so they're fully pg_dump'd (custom format) every day to leaf_YYYY-MM-DD.dump. sensor_data and TimescaleDB's internal chunk/catalog tables are excluded.
2. sensor_data — exported per UTC day via \copy to sensor_data_YYYY-MM-DD.csv.gz. The last ROLLING_DAYS days are re-exported (overwritten) on every run to catch late-arriving readings; older files are write-once.

The image is read-only against the database (leaf_backup_user, member of the backup_readers role created by deploy/deploy.sql) and runs once per invocation — schedule it with a Kubernetes CronJob (see docker/backup/cronjob.example.yaml) or any host cron running docker run.

docker build -t leaf-backup docker/backup

docker run --rm \
  -e PGHOST=... -e PGUSER=leaf_backup_user -e PGPASSWORD=... -e PGDATABASE=leaf \
  -e ROLLING_DAYS=3 -e KEEP_DUMPS=14 \
  -v /path/to/backups:/backups \
  leaf-backup

Or, using deploy/deploy.py (builds/pushes via build-backup, runs once via backup):

python3 deploy/deploy.py build-backup   # build & push to the registry

export PGHOST=... PGUSER=leaf_backup_user PGPASSWORD=... PGDATABASE=leaf
export BACKUP_DIR=/path/to/backups      # default: ./backups
python3 deploy/deploy.py backup

Restore:

# 1. Recreate the schema (also recreates the sensor_data hypertable)
python3 deploy/deploy.py schema

# 2. Restore operational tables
pg_restore --data-only --disable-triggers -d <db> leaf_YYYY-MM-DD.dump

# 3. Re-import sensor_data for each day
zcat sensor_data_YYYY-MM-DD.csv.gz | psql -d <db> -c "\copy sensor_data FROM STDIN WITH (FORMAT csv, HEADER true)"

The backup image pins its pg_dump/pg_restore version to the TimescaleDB version this project targets (timescale/timescaledb:2.17.2-pg16) — custom-format dumps aren't readable by an older pg_restore. To back up a different Postgres major version, change the FROM tag in docker/backup/Dockerfile and rebuild. Before dumping anything, backup.sh checks that the server's major version matches its bundled pg_dump and exits with an error (without writing any files) if they've drifted apart.

Deployment

deploy/deploy.py is a helper script for building and running the portal in production. It requires no extra dependencies beyond Docker (and psql for remote schema application).

python3 deploy/deploy.py <command>

schema requires passwords for the PostgreSQL service accounts it creates — never use defaults:

export LEAF_PORTAL_PASSWORD=...   # leaf_portal_user  (portal, read+write)
export LEAF_GRAFANA_PASSWORD=...  # leaf_grafana_user (Grafana, read-only)
export LEAF_API_PASSWORD=...      # leaf_api_user     (external API access)
export LEAF_BACKUP_PASSWORD=...   # leaf_backup_user  (backup job, read-only)
export LEAF_VERNEMQ_PASSWORD=...  # leaf_vernemq_user (VerneMQ Postgres-auth, SELECT-only on vmq_auth_acl)
export LEAF_NODERED_AUTH_PASSWORD=... # leaf_nodered_auth_user (Node-RED admin login, column-limited SELECT on user_account)

Typical production flow:

# 1. Apply the schema (once, or after schema changes)
python3 deploy/deploy.py schema

# 2. Start the portal
python3 deploy/deploy.py run

Run build only when cutting a new release.

Page routes