scalio-selftest

Test mode of the Scalio apps (scalio-docs ADR 0053, concepts/test-mode.md): on our test instances (never dev, demo or production) the app checks itself every night from the outside (API suite + browser smoke), keeps the history, compares each run with the one before, reports to Scalio Monitor and shows everything on the page «Tests» (@scalio-cloud/ui). Modelled on the Nexus self-check (backend/e2e_selfcheck.py, backend/test.py, backend/ui_smoke.py).

import scalio_selftest as sst
from scalio_selftest import common

async def basics(ctx):
    await common.basics(ctx)
    await common.sign_in(ctx)
    await common.setup_closed(ctx)
    await common.rights(ctx, [common.Rule("GET", "/api/users", {"admin"})])

SELFTEST = sst.SelfTest(
    "citadel", db, [sst.Suite("Basis", basics)],
    accounts=[sst.account("citadel", "admin"), sst.account("citadel", "operator")],
    upsert_account=upsert, remove_accounts=remove_all,          # the app stores its users itself
    ui=sst.UiSmoke(pages=[sst.Page("/", "Übersicht"), sst.Page("/settings", "Einstellungen")]),
)

app.include_router(make_router(SELFTEST, require_admin))       # scalio_selftest.router, needs [fastapi]
await SELFTEST.startup()                                         # API and worker start: accounts on / off
JOBS += [("citadel_selftest", 30, SELFTEST.tick)]                # worker, under the job lease

Switches

Pieces

• gate: the two switches above. The UI can lower FULL to SAFE, never raise it.
• accounts: one hidden test account per role (e2e-<role>@<app>.e2e.scaliocloud.com, no MX), fresh password on every start, kept only in the app's database, removed when the test mode is off. totp() for test accounts with MFA.
• store: MongoStore(db) (collections e2e_state, e2e_runs) or SqlStore(engine) (table e2e_docs, SqlStore.DDL, for Kontor). SelfTest takes a Motor database or a store.
• runner: SelfTest (history of 30 runs, progress, request, config), Context with check(), client(role), defer() for clean-up, objects named with ctx.prefix (e2e-<run>).
• common: checks every app runs: health, version, headers, sign-in per role, wrong password, rights matrix, first-run setup closed, reads without errors.
• ui: browser smoke over E2E_BROWSER_WS in German (light) and English (dark): sign-in, pages render, no error page, no sharp s, console errors as WARN.
• router: /api/e2e/status (public, verdict only), health, runs, progress, config, run, cancel (admin), trigger (deploy token).
• monitor: heartbeat POST /ingest/heartbeat/<key>?status=ok|fail.

Result format (same as Nexus): {section, name, mode, status, detail} with PASS, FAIL, WARN, SKIP.