hop3-testing
Testing framework for Hop3 deployment validation.
Overview
hop3-testing provides utilities and fixtures for testing Hop3 deployments. It supports running tests against Docker containers or remote servers, with a catalog of test applications covering various languages and frameworks.
Features
- Multiple targets: Test against Docker containers or remote SSH servers
- App catalog: Pre-built test applications for various languages
- Deployment sessions: Automated deploy/verify/cleanup workflow
- State checks: after the validations, an app fails if it wrote into its
src/while running (every deploy replacessrc/, so that data is lost at the next one; the failure lists the directories to declare as[[volumes]]). The unchanged source is then deployed again and every validation runs a second time - pytest fixtures: Integration with pytest for E2E testing
- Category filtering: Run tests by language or framework
Installation
pip install hop3-testing
Quick Start
# List available tests
hop3-test list
# Run system tests on Docker
hop3-test run --docker
# Test specific apps
hop3-test apps 010-flask-pip-wsgi
# Test against remote server
hop3-test apps --host server.example.com
# Run CI tests
hop3-test ci
Commands
| Command | Description |
|---|---|
hop3-test run |
Deploy Hop3 and run the catalog (system is a deprecated alias) |
hop3-test run --reuse |
Test against an existing deployment (skip deploy) |
hop3-test list |
List available tests (--show NAME for one test's details) |
hop3-test run --provider hetzner --images ... |
E2E across cloud OS images (Hetzner) |
hop3-test upgrade-chain |
Install a baseline release on a fresh box, then upgrade in-place through a version chain |
hop3-test why <run-id> |
Replay a saved diagnostic bundle for a failed run |
Test profile (fast/CI/full) is selected with --mode on run, not a separate
subcommand: hop3-test run --docker --mode ci.
Upgrade chain
hop3-test upgrade-chain validates that a running server survives a chain of in-place upgrades. Each hop is a git ref, installed by that version's own hop3-deploy-server (checked out into a worktree and run via uv run), on a fresh box; every hop after the first is an in-place update, and each is asserted to come back healthy with a readable schema.
The first hop also seeds the state an upgrade must carry: an app (010-flask-pip-wsgi), deployed once and never again, and a plain account. After every hop the app must still serve its page, the admin must still list the accounts, and the account must do exactly what it could on the first hop (get a token, read its app's environment, list the accounts), probed by doing each. That is how the chain checks a migration such as ADR 061's roles, which must turn every non-admin into a deployer. Before a release, run it from the last release: --chain <last-tag>,local.
# Fresh Docker container: 0.7.5 → current tree
hop3-test upgrade-chain --docker
# Fresh Hetzner VPS (needs HETZNER_API_TOKEN + HETZNER_SERVER_ID)
hop3-test upgrade-chain --provider hetzner --image ubuntu-24.04
# Custom chain (release tags + `local` for the current tree)
hop3-test upgrade-chain --provider hetzner --chain 0.6.2,0.7.5,local
On Docker, this checkout's deployer drives every hop: the deployers of releases up to 0.7.5 bind fixed host ports and check the server at localhost:8000, which a shared or remote Docker host cannot give them. The first hop may be a release: the deployer copies that release's tree to the box (hop3-deploy-server --source-tree) and that release's own installer installs it, while the container setup (supervisor in place of systemd) is the current one. Later hops must be local, and the release's installer must accept --path and --operator-email, so 0.6.x baselines run with --provider hetzner only.
Cheapest smoke — exercises the whole mechanism (fresh install → in-place upgrade → assertions) with no old-version/worktree variable:
hop3-test upgrade-chain --docker --chain local,local
--host <server> is accepted but warns: it targets an existing server, not the clean slate an upgrade chain assumes. 0.6.0 is not a viable baseline (its hop3-rootd can't start) and is excluded from the default chain.
Common Options
| Option | Description |
|---|---|
-v, --verbose |
Verbose output |
--fail-fast |
Stop on first failure |
--keep |
Keep apps deployed after testing |
--no-redeploy |
Deploy each app once, skipping the second deploy and validation pass |
--docker |
Use Docker target |
--host HOST |
Remote server hostname |
--screenshots |
Photograph each catalog recipe signed in, into its screenshots/ (a server only) |
--screenshots-to DIR |
Also copy every image into DIR (implies --screenshots) |
Screenshots, and the sign-ins only a browser can check
--screenshots drives a real browser to each passed catalog recipe before it is destroyed: its sign-in page, then its first page once signed in with the app's admin credential. The images land in the recipe's screenshots/ in the catalog checkout, which the catalog publishes. The browser reaches the app over HTTPS at its test hostname, resolved to the server, so it needs a server target; on Docker the flag is refused. Playwright is loaded only for this:
uv run --with playwright playwright install chromium # once
uv run --with playwright hop3-test run --host <server> --recipes published --screenshots
Some apps build their sign-in form in JavaScript, so their check.py verifies everything else and leaves the sign-in to a browser (SIGN-IN VERIFIED BY BROWSER). With --screenshots, a browser that cannot sign in fails the test. Without it, the test is reported as not verified: listed by name in the summary, stored with the status unverified, and not counted as a failure, since nothing found the app broken. A release run uses --screenshots, so those apps count only once a browser has signed in.
Features follow the apps
The server is installed with what the selected apps declare: their addons ([[addons]]) and the runtime their builder needs (nix for a Nix-built app, docker for a Docker-built one). --with adds to that. Nothing is installed by default. The Docker target cannot run Docker, so Docker-built apps are left out there, each with the reason; test them against a server with --host.
A server deploy that warns fails
Every ⚠ line the installer or the deployer prints fails the server deploy, unless it matches TOLERATED_DEPLOY_WARNINGS in targets/helpers.py, where each entry says why continuing is safe. A warning-and-continue is the failure a review misses and a red run does not: two deployer bugs and an install failing on every Ubuntu 24.04 server hid behind warnings every log carried. Fix the warning, or add it with its justification. Old releases' deployers, driven by upgrade-chain, are not gated. See notes/installer-fail-loud-audit.md.
Architecture
hop3-testing/
├── src/hop3_testing/
│ ├── cli/ # CLI commands
│ │ └── commands/ # Click command implementations
│ ├── catalog/ # Test discovery
│ │ ├── scanner.py # Discovers test.toml files
│ │ └── models.py # TestDefinition, Category
│ ├── apps/
│ │ ├── catalog.py # AppSource dataclass
│ │ └── deployment.py # DeploymentSession
│ ├── targets/
│ │ ├── base.py # DeploymentTarget ABC
│ │ ├── docker.py # DockerTarget
│ │ └── remote.py # RemoteTarget
│ └── results/ # Result storage and reporting
└── tests/
Test Categories
| Category | Languages/Frameworks |
|---|---|
python |
Flask, FastAPI, Django |
nodejs |
Express, Fastify |
ruby |
Sinatra, Rails |
go |
Fiber, Gin |
rust |
Actix-web, Axum |
static |
HTML, Hugo, Jekyll |
Cloud Runs & the Image Sweep
Run full E2E tests across cloud OS images with hop3-test run --provider hetzner --images .... Each image is a full hop3-test run --provider hetzner (provision a
fresh box → deploy → test → persist), so a cloud run shares run's lexicon:
positional app names, --from, --branch, --with. Requires HETZNER_API_TOKEN
and HETZNER_SERVER_ID (a dedicated throwaway box). A rebuild wipes the box, so
it refuses any server that does not carry the Hetzner label
hop3-throwaway=true; set it once, on that server only:
hcloud server add-label $HETZNER_SERVER_ID hop3-throwaway=true.
# List available images
hop3-test run --list-images
# Single distribution (a sweep-of-one)
hop3-test run --provider hetzner --image ubuntu-24.04 apps/test-apps-procfile
# Across multiple distributions
hop3-test run --provider hetzner --images ubuntu-24.04,debian-13,fedora-42
# From PyPI instead of local code
hop3-test run --provider hetzner --from pypi --images all
Cloud Run Options (run --provider hetzner)
| Option | Description |
|---|---|
--image IMAGE |
Single OS image — a sweep-of-one (e.g. ubuntu-24.04) |
--images LIST |
Comma-separated images or all |
--list-images |
List available OS images |
--from {local,git,pypi} |
Install source (same as run; default: local) |
--branch BRANCH |
Git branch (with --from git; default: devel) |
--with FEATURES |
Extra server features (repeatable or comma-separated) |
-x, --fail-fast |
Stop on the first failing image |
--continue-on-failure |
Don't stop on first failure (multi-distro) |
Development
# Run tests
uv run pytest tests/ -v
# Lint and format
uv run ruff check src/
uv run ruff format src/
Documentation
Related Packages
- hop3-server - The server being tested
- hop3-cli - CLI used for deployments in tests
License
Apache-2.0 - Copyright (c) 2024-2026, Abilian SAS
Metadata
Release files for hop3-testing 0.8.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| hop3_testing-0.8.1.tar.gz | 227.3 kB | Without provenance | Details for hop3_testing-0.8.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| hop3_testing-0.8.1-py3-none-any.whl | Python 3 | none | any | Details for hop3_testing-0.8.1-py3-none-any.whl |
Total release size: 497.6 kB
Release files / hop3_testing-0.8.1.tar.gz
| Download URL | hop3_testing-0.8.1.tar.gz |
|---|---|
| Size | 227.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b691e5444c2afbb9dd6d34dfb31b600d8de798579ce04b02445e70641d0f866a
|
|
BLAKE2b-256 checksum How to use checksums |
59a05ca396a9add22296947c51608238e0f0fe12f3b3f8782bceaea9dd0fa40b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.14
|
Release files / hop3_testing-0.8.1-py3-none-any.whl
| Download URL | hop3_testing-0.8.1-py3-none-any.whl |
|---|---|
| Size | 270.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e4f99e3d56106bbffb062e72a8334b523b5aef339df3f7f6584f8ca7e7067a5c
|
|
BLAKE2b-256 checksum How to use checksums |
789e12c99ea4132133f723e7c59c98956409c002f99647ff96b40caadbb0f914
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.14
|
Security for hop3-testing 0.8.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.