Skip to main content

hop3-testing

Testing framework for Hop3 deployment validation.

Overview

hop3-testing provides utilities and fixtures for testing Hop3 deployments. It supports running tests against Docker containers or remote servers, with a catalog of test applications covering various languages and frameworks.

Features

  • Multiple targets: Test against Docker containers or remote SSH servers
  • App catalog: Pre-built test applications for various languages
  • Deployment sessions: Automated deploy/verify/cleanup workflow
  • State checks: after the validations, an app fails if it wrote into its src/ while running (every deploy replaces src/, so that data is lost at the next one; the failure lists the directories to declare as [[volumes]]). The unchanged source is then deployed again and every validation runs a second time
  • pytest fixtures: Integration with pytest for E2E testing
  • Category filtering: Run tests by language or framework

Installation

pip install hop3-testing

Quick Start

# List available tests
hop3-test list

# Run system tests on Docker
hop3-test run --docker

# Test specific apps
hop3-test apps 010-flask-pip-wsgi

# Test against remote server
hop3-test apps --host server.example.com

# Run CI tests
hop3-test ci

Commands

Command Description
hop3-test run Deploy Hop3 and run the catalog (system is a deprecated alias)
hop3-test run --reuse Test against an existing deployment (skip deploy)
hop3-test list List available tests (--show NAME for one test's details)
hop3-test run --provider hetzner --images ... E2E across cloud OS images (Hetzner)
hop3-test upgrade-chain Install a baseline release on a fresh box, then upgrade in-place through a version chain
hop3-test why <run-id> Replay a saved diagnostic bundle for a failed run

Test profile (fast/CI/full) is selected with --mode on run, not a separate subcommand: hop3-test run --docker --mode ci.

Upgrade chain

hop3-test upgrade-chain validates that a running server survives a chain of in-place upgrades. Each hop is a git ref, installed by that version's own hop3-deploy-server (checked out into a worktree and run via uv run), on a fresh box; every hop after the first is an in-place update, and each is asserted to come back healthy with a readable schema.

The first hop also seeds the state an upgrade must carry: an app (010-flask-pip-wsgi), deployed once and never again, and a plain account. After every hop the app must still serve its page, the admin must still list the accounts, and the account must do exactly what it could on the first hop (get a token, read its app's environment, list the accounts), probed by doing each. That is how the chain checks a migration such as ADR 061's roles, which must turn every non-admin into a deployer. Before a release, run it from the last release: --chain <last-tag>,local.

# Fresh Docker container: 0.6.2 → current tree
hop3-test upgrade-chain --docker

# Fresh Hetzner VPS (needs HETZNER_API_TOKEN + HETZNER_SERVER_ID)
hop3-test upgrade-chain --provider hetzner --image ubuntu-24.04

# Custom chain (release tags + `local` for the current tree)
hop3-test upgrade-chain --docker --chain 0.6.2,local

Cheapest smoke — exercises the whole mechanism (fresh install → in-place upgrade → assertions) with no old-version/worktree variable:

hop3-test upgrade-chain --docker --chain local,local

--host <server> is accepted but warns: it targets an existing server, not the clean slate an upgrade chain assumes. 0.6.0 is not a viable baseline (its hop3-rootd can't start) and is excluded from the default chain.

Common Options

Option Description
-v, --verbose Verbose output
--fail-fast Stop on first failure
--keep Keep apps deployed after testing
--no-redeploy Deploy each app once, skipping the second deploy and validation pass
--docker Use Docker target
--host HOST Remote server hostname
--screenshots Photograph each catalog recipe signed in, into its screenshots/ (a server only)
--screenshots-to DIR Also copy every image into DIR (implies --screenshots)

Screenshots, and the sign-ins only a browser can check

--screenshots drives a real browser to each passed catalog recipe before it is destroyed: its sign-in page, then its first page once signed in with the app's admin credential. The images land in the recipe's screenshots/ in the catalog checkout, which the catalog publishes. The browser reaches the app over HTTPS at its test hostname, resolved to the server, so it needs a server target; on Docker the flag is refused. Playwright is loaded only for this:

uv run --with playwright playwright install chromium     # once
uv run --with playwright hop3-test run --host <server> --recipes published --screenshots

Some apps build their sign-in form in JavaScript, so their check.py verifies everything else and leaves the sign-in to a browser (SIGN-IN VERIFIED BY BROWSER). With --screenshots, a browser that cannot sign in fails the test. Without it, the test is reported as not verified: listed by name in the summary, stored with the status unverified, and not counted as a failure, since nothing found the app broken. A release run uses --screenshots, so those apps count only once a browser has signed in.

Features follow the apps

The server is installed with what the selected apps declare: their addons ([[addons]]) and the runtime their builder needs (nix for a Nix-built app, docker for a Docker-built one). --with adds to that. Nothing is installed by default. The Docker target cannot run Docker, so Docker-built apps are left out there, each with the reason; test them against a server with --host.

A server deploy that warns fails

Every ⚠ line the installer or the deployer prints fails the server deploy, unless it matches TOLERATED_DEPLOY_WARNINGS in targets/helpers.py, where each entry says why continuing is safe. A warning-and-continue is the failure a review misses and a red run does not: two deployer bugs and an install failing on every Ubuntu 24.04 server hid behind warnings every log carried. Fix the warning, or add it with its justification. Old releases' deployers, driven by upgrade-chain, are not gated. See notes/installer-fail-loud-audit.md.

Architecture

hop3-testing/
├── src/hop3_testing/
│   ├── cli/              # CLI commands
│   │   └── commands/     # Click command implementations
│   ├── catalog/          # Test discovery
│   │   ├── scanner.py    # Discovers test.toml files
│   │   └── models.py     # TestDefinition, Category
│   ├── apps/
│   │   ├── catalog.py    # AppSource dataclass
│   │   └── deployment.py # DeploymentSession
│   ├── targets/
│   │   ├── base.py       # DeploymentTarget ABC
│   │   ├── docker.py     # DockerTarget
│   │   └── remote.py     # RemoteTarget
│   └── results/          # Result storage and reporting
└── tests/

Test Categories

Category Languages/Frameworks
python Flask, FastAPI, Django
nodejs Express, Fastify
ruby Sinatra, Rails
go Fiber, Gin
rust Actix-web, Axum
static HTML, Hugo, Jekyll

Cloud Runs & the Image Sweep

Run full E2E tests across cloud OS images with hop3-test run --provider hetzner --images .... Each image is a full hop3-test run --provider hetzner (provision a fresh box → deploy → test → persist), so a cloud run shares run's lexicon: positional app names, --from, --branch, --with. Requires HETZNER_API_TOKEN and HETZNER_SERVER_ID (a dedicated throwaway box). A rebuild wipes the box, so it refuses any server that does not carry the Hetzner label hop3-throwaway=true; set it once, on that server only: hcloud server add-label $HETZNER_SERVER_ID hop3-throwaway=true.

# List available images
hop3-test run --list-images

# Single distribution (a sweep-of-one)
hop3-test run --provider hetzner --image ubuntu-24.04 apps/test-apps-procfile

# Across multiple distributions
hop3-test run --provider hetzner --images ubuntu-24.04,debian-13,fedora-42

# From PyPI instead of local code
hop3-test run --provider hetzner --from pypi --images all

Cloud Run Options (run --provider hetzner)

Option Description
--image IMAGE Single OS image — a sweep-of-one (e.g. ubuntu-24.04)
--images LIST Comma-separated images or all
--list-images List available OS images
--from {local,git,pypi} Install source (same as run; default: local)
--branch BRANCH Git branch (with --from git; default: devel)
--with FEATURES Extra server features (repeatable or comma-separated)
-x, --fail-fast Stop on the first failing image
--continue-on-failure Don't stop on first failure (multi-distro)

Development

# Run tests
uv run pytest tests/ -v

# Lint and format
uv run ruff check src/
uv run ruff format src/

Documentation

License

Apache-2.0 - Copyright (c) 2024-2026, Abilian SAS

Metadata

Release files for hop3-testing 0.8.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hop3-testing 0.8.0
File Size Uploaded Provenance Actions
hop3_testing-0.8.0.tar.gz 222.9 kB Without provenance Details for hop3_testing-0.8.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for hop3-testing 0.8.0
File Interpreter ABI Platform Actions
hop3_testing-0.8.0-py3-none-any.whl Python 3 none any Details for hop3_testing-0.8.0-py3-none-any.whl

Total release size: 488.9 kB

Release files / hop3_testing-0.8.0.tar.gz

Download URL hop3_testing-0.8.0.tar.gz
Size 222.9 kB
Tags Source
SHA-256 checksum
How to use checksums
86a93b2218cae78dc86c061ed8182a5d50da7b890a2c8806805a4373407d0ad1
BLAKE2b-256 checksum
How to use checksums
85f528662f49eb807839ca4b421d223a6a1b2509b7b4bd30ccd466fef035070c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.13

Release files / hop3_testing-0.8.0-py3-none-any.whl

Download URL hop3_testing-0.8.0-py3-none-any.whl
Size 266.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
81e294e646b48a9b2a0b19611e18e27832539621df3105102855234a968129e0
BLAKE2b-256 checksum
How to use checksums
a11742b05ad0579b65ef12cd36f77e345423912c1e3bbd844cacf2dd09a688fa
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.13

Security for hop3-testing 0.8.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page