esys-ci-host 0.5.1

Library and CLI to prepare Linux and Windows hosts as GitLab CI runners

esys-ci-host

Library and CLI to prepare Linux VPS hosts as GitLab CI runners (Docker, privileged DinD, runner registration).

Extracted from the CI-host portions of esysdox-ops so LibESys superbuild and other tooling can depend on a focused package.

Install

pip install esys-ci-host

On a VPS, prefer the dedicated virtualenv used by remote mode (avoids apt/pip conflicts):

sudo apt-get update
sudo apt-get install -y python3-venv python3-pip python3.12-venv   # match your python3 -V
sudo mkdir -p /opt/esys-ci-host
sudo rm -rf /opt/esys-ci-host/venv
sudo python3 -m venv /opt/esys-ci-host/venv
sudo /opt/esys-ci-host/venv/bin/python -m ensurepip --upgrade
sudo /opt/esys-ci-host/venv/bin/python -m pip install esys-ci-host
sudo ln -sf /opt/esys-ci-host/venv/bin/esys-ci-host /usr/local/bin/esys-ci-host

Remote commands (esys-ci-host remote ...) create and use this venv automatically.

Updating

Upgrade esys-ci-host without touching pip paths manually. The command detects how the currently running CLI was installed and upgrades that environment.

Install type	Detected when	Upgrade method
Managed VPS	Package under /opt/esys-ci-host/venv	sudo + refresh /usr/local/bin/esys-ci-host
Virtualenv	CLI runs inside a Python venv	python -m pip install --upgrade in that venv
User install	Package under ~/.local	pip install --user --upgrade

On the VPS (as admin):

esys-ci-host update                  # latest from PyPI
esys-ci-host update --version 0.3.6  # specific version
esys-ci-host update --dry-run        # show detected install and planned pip command

Do not use sudo pip install on the VPS — that hits system Python and can conflict with apt.

From your laptop:

esys-ci-host remote --host 161.97.147.246 --user esysadmin --identity-file ~/.ssh/id_ed25519 update
esys-ci-host remote --host ... --user esysadmin --identity-file ... update --version 0.3.6

Option	Purpose
--version VERSION	Pin the PyPI version (default: latest)
--wheel PATH	Install from a local wheel instead of PyPI
--dry-run	Print detected install layout without upgrading

On Windows, update cannot replace esys-ci-host.exe while it is running. The command schedules pip to run after the current process exits. You can also run python -m esys_ci_host update from the same virtualenv.

Remote update runs update on the VPS. On older installs without the update subcommand, it falls back to upgrading the managed virtualenv directly.

Quick start

On a fresh Ubuntu or Debian VPS (Docker executor for LibESys CI):

sudo esys-ci-host host bootstrap --yes --admin-user deploy --ssh-key-file ~/.ssh/id_ed25519.pub
# from your laptop: ssh deploy@<vps-ip>   (confirm login works)
sudo esys-ci-host host harden --yes
sudo esys-ci-host runner prepare --yes
esys-ci-host runner register --url https://gitlab.example.com --profile ci
esys-ci-host doctor

host bootstrap creates the admin user, installs OpenSSH, installs your SSH public key (or generates one under /etc/esys-ci-host/), verifies key setup, and enables UFW with SSH allowed. Root SSH stays enabled until you run host harden after confirming deploy login from your machine. Pass --ssh-key-file ~/.ssh/id_ed25519.pub or set ESYS_CI_HOST_ADMIN_SSH_PUBLIC_KEY / ESYS_CI_HOST_ADMIN_SSH_PUBLIC_KEY_FILE. Use --disable-root-ssh only if you accept lockout risk without external login verification.

The admin user has no login password (SSH key only). Bootstrap grants passwordless sudo via /etc/sudoers.d/99-esys-ci-host-<admin-user> so commands like sudo esys-ci-host host harden work after key login.

Set ESYS_CI_HOST_RUNNER_TOKEN instead of passing --token when registering.

Remote configuration (from your laptop)

Run the same host and runner steps over SSH without logging in manually first. Remote mode installs esys-ci-host into /opt/esys-ci-host/venv on the VPS (from PyPI or a local wheel), links /usr/local/bin/esys-ci-host, uploads your local SSH public key for the admin user, and runs the usual commands there.

Interactive (prompts for the root password when no key is given):

esys-ci-host remote --host 203.0.113.10 host bootstrap --yes --admin-user deploy
ssh -i ~/.ssh/id_ed25519 deploy@203.0.113.10   # verify login (admin has no password)
esys-ci-host remote --host 203.0.113.10 --user deploy --identity-file ~/.ssh/id_ed25519 host harden --yes
esys-ci-host remote --host 203.0.113.10 --user deploy --identity-file ~/.ssh/id_ed25519 runner prepare --yes

After a successful remote bootstrap, the tool can offer to write a managed ~/.ssh/config entry. It suggests a short name (when --save-host is set, that alias is used; otherwise for an IP like 161.97.147.246, default is {admin-user}-vps, e.g. esysadmin-vps; for runner01.example.com, default is runner01) and lets you change it before saving. Then ssh esysadmin-vps (or your chosen name) works. Omit --setup-ssh-config to be prompted; use --setup-ssh-config to apply without prompts, --ssh-config-host NAME to set the alias directly, or --skip-ssh-config to disable the flow.

Remote mode reads basic OpenSSH Host defaults from ~/.ssh/config (user, port, IdentityFile) when you omit CLI flags. It does not expand Include files or honor IdentityAgent pipes — load keys into Pageant or ssh-agent and omit --identity-file, or pass --identity-file with ESYS_CI_HOST_SSH_IDENTITY_PASSPHRASE for encrypted keys.

Non-interactive (CI or scripts — password via flag or env):

export ESYS_CI_HOST_SSH_PASSWORD='...'
esys-ci-host remote --host 203.0.113.10 --non-interactive host bootstrap --yes --admin-user deploy

Option	Purpose
--host HOST	Remote VPS hostname or IP (required)
--user USER	SSH login user (default: root)
--password PASS	SSH password (ESYS_CI_HOST_SSH_PASSWORD)
--identity-file PATH	SSH private key (ESYS_CI_HOST_SSH_IDENTITY_FILE)
ESYS_CI_HOST_SSH_IDENTITY_PASSPHRASE	Passphrase for an encrypted --identity-file
--non-interactive	Fail instead of prompting for a password
--wheel PATH	Install from a local wheel instead of PyPI (Windows: use full path, not *)
--no-version-check	Skip local vs remote version/build warnings on SSH connect
--ssh-key-file PATH	Local admin public key (default: ~/.ssh/id_ed25519.pub)

Prefer --identity-file over --password after the first bootstrap. For encrypted keys, set ESYS_CI_HOST_SSH_IDENTITY_PASSPHRASE or allow an interactive passphrase prompt. With no --identity-file, authentication uses ssh-agent or Pageant (no server password required). Avoid putting passwords on the command line in shared environments.

Version checks: esys-ci-host version and esys-ci-host remote --host HOST version show the package version and a short build fingerprint (SHA-256 of installed Python sources). Other remote commands warn when local and remote differ; use --strict on remote version to fail in scripts.

Commands still run on the VPS when you SSH in directly (host bootstrap, runner prepare, etc.) — both models are supported.

Host bootstrap

Option	Purpose
--admin-user NAME	Non-root login account with sudo (default: esysadmin)
--ssh-key KEY	OpenSSH public key line for admin login
--ssh-key-file PATH	File containing the public key
--ssh-port PORT	SSH port used for localhost login verification (default: 22)
--disable-root-ssh	Disable root SSH during bootstrap (default: keep root until host harden)
--yes	Required to apply host changes
--dry-run	Print steps without mutating the host

Admin login is SSH key only (no account password). Bootstrap writes a sudoers drop-in so the admin user can run sudo without a password.

If no key is supplied, an ed25519 key pair is generated at /etc/esys-ci-host/<admin-user>-id_ed25519 — copy the private key securely before closing your root session.

After bootstrap: SSH in as the admin user from your machine, then run sudo esys-ci-host host harden --yes to disable root login.

Runner registration and executors

The GitLab executor is chosen at registration time (runner register), not during runner install or runner prepare.

Option	Purpose
--executor shell	Run jobs directly on the host (default)
--executor docker	Run jobs in containers (requires runner prepare)
--profile ci	LibESys CI preset: docker executor, privileged DinD, tags docker-privileged, and a sensible default image
--docker-image IMAGE	Override the runner’s default image (fallback only; see note below)
--docker-privileged	Privileged docker executor (enabled automatically by --profile ci)
--tags TAGS	Comma-separated runner tags (e.g. linux,docker)

Default image vs job image: GitLab Runner’s --docker-image (at register time) is only the fallback written into config.toml. Each CI job uses its own image: from .gitlab-ci.yml when present — you do not need to pass --docker-image for every image your pipeline uses. With --profile ci, a default (python:3.11-bookworm) is applied automatically; override it only if you want a different fallback for jobs that omit image:.

Shell executor (simple jobs on the host):

sudo esys-ci-host runner install --yes
esys-ci-host runner register --url https://gitlab.example.com --executor shell --tags shell

Docker executor (explicit; run prepare first). GitLab Runner requires a default image for non-interactive docker registration, so pass --docker-image here unless you use --profile ci:

sudo esys-ci-host runner prepare --yes
esys-ci-host runner register \
  --url https://gitlab.example.com \
  --executor docker \
  --docker-image python:3.11-bookworm \
  --docker-privileged \
  --tags docker-privileged

LibESys CI profile (recommended; includes docker executor defaults — no need to pass --docker-image unless overriding the fallback):

runner prepare creates the gitlab-admin group, adds the bootstrap admin (and gitlab-runner for service access), and makes /etc/gitlab-runner group-writable so runner register can patch config.toml without sudo for every read/write. Register and systemctl still use sudo when needed.

sudo esys-ci-host runner prepare --yes
esys-ci-host runner register --url https://gitlab.example.com --profile ci

Windows Server CI runner (shell executor, tags windows,libesys; run in an elevated PowerShell or cmd). runner prepare installs Git, Python 3.11, Conan 2, and Visual Studio 2022 Build Tools with the MSVC 14.4 toolset (Conan msvc2022-x64 / compiler.version=194). If you only have VS 18 installed, prepare adds VS 2022 Build Tools side-by-side (VS 18 uses MSVC 14.5). Re-run prepare on an existing VS 2022 host to upgrade a 14.3-only install.

pip install esys-ci-host
esys-ci-host runner prepare --yes
esys-ci-host runner register --url https://gitlab.example.com --profile ci --token <token>

Set ESYS_CI_HOST_RUNNER_TOKEN instead of --token. Optionally set ESYS_CI_HOST_RUNNER_USER_PASSWORD for the gitlab-runner service account; otherwise a password is generated once and saved to C:\ProgramData\esys-ci-host\gitlab-runner-account.json (Administrators only — retrieve with Get-Content in an elevated PowerShell). Check esys-ci-host runner status for the path. Shell jobs should call C:\Users\gitlab-runner\.esys-ci-host\ci-env.cmd (written by prepare) or set pre_build_script in GitLab Runner config.toml to load MSVC and Conan on PATH.

Use --dry-run on runner register to print the gitlab-runner register command without executing it.

Runner operations

Command	Purpose
runner list	Show runners in config.toml (add --json for machine-readable output)
runner add	Alias for runner register — register a new runner with GitLab
runner start / runner stop / runner restart	Control the local GitLab Runner service
runner remove --name NAME --yes	Unregister one runner from GitLab
runner remove --all --yes	Unregister every configured runner
runner status	Install, service, and esys-ci-host registration metadata

The same commands work remotely over SSH (prefix with remote --host HOST --user USER --identity-file …):

esys-ci-host remote --host 203.0.113.10 --user deploy --identity-file ~/.ssh/id_ed25519 runner list
esys-ci-host remote --host 203.0.113.10 --user deploy --identity-file ~/.ssh/id_ed25519 runner stop
esys-ci-host remote --host 203.0.113.10 --user deploy --identity-file ~/.ssh/id_ed25519 runner remove --name "ci-runner" --yes

esys-ci-host runner list
esys-ci-host runner stop
esys-ci-host runner start
esys-ci-host runner remove --name "esys-ci-host docker runner" --yes

Artifactory registry host

Prepare a dedicated VPS for Artifactory C++ CE (Conan packages for LibESys CI). The container binds to localhost only (127.0.0.1:8081 / 8082); artifactory proxy (or deploy) exposes it at https://artifactory.libesys.org with automatic Let's Encrypt TLS via Caddy.

One-shot deploy (prepare + bootstrap + HTTPS) after host bootstrap:

sudo esys-ci-host artifactory deploy --yes
esys-ci-host doctor

Or step by step:

sudo esys-ci-host artifactory prepare --yes
sudo esys-ci-host artifactory bootstrap --yes
sudo esys-ci-host artifactory proxy --yes   # Caddy + Let's Encrypt HTTPS
esys-ci-host doctor

From your laptop (after host bootstrap / host harden):

esys-ci-host remote --host artifactory.libesys.org --user esysadmin --identity-file ~/.ssh/id_ed25519 \
  artifactory deploy --yes

Full automated flow from your laptop (DNS artifactory.libesys.org → VPS IP required for HTTPS):

# Build wheel locally until a release is on PyPI:
pip wheel . -w dist/

esys-ci-host remote --host artifactory.libesys.org --user root --identity-file ~/.ssh/id_ed25519 \
  host bootstrap --yes --setup-ssh-config

esys-ci-host remote --host artifactory.libesys.org --user esysadmin --identity-file ~/.ssh/id_ed25519 \
  host harden --yes

esys-ci-host remote --host artifactory.libesys.org --user esysadmin --identity-file ~/.ssh/id_ed25519 \
  --wheel dist/esys_ci_host-*.whl artifactory deploy --yes

deploy runs prepare (when needed), bootstrap, and HTTPS proxy in one command. Pass --save-host artifactory (default on remote artifactory deploy) to store deploy credentials and Conan remote URL locally under ~/.esys-ci-host/hosts/.

Private image registry (CI Docker images)

Prepare a local OCI registry on the CI runner VPS (registry:2 on 127.0.0.1:5000) so GitLab jobs can push and pull libesys/superbuild images without GitLab Container Registry storage costs.

sudo esys-ci-host image-registry prepare --yes
esys-ci-host doctor

From your laptop (after runner prepare on the VPS):

esys-ci-host remote --host esysdocker --user esysadmin --identity-file ~/.ssh/id_ed25519 \
  image-registry prepare --yes

esys-ci-host remote --host esysdocker doctor

Credentials are saved on the VPS at /etc/esys-ci-host/image-registry/bootstrap.json (mode 640, group esys-ci-host). Members include the bootstrap admin and gitlab-runner so CI jobs can read registry credentials without sudo. Set ESYS_CI_HOST_IMAGE_REGISTRY_PASSWORD to pin the htpasswd password instead of generating one.

Pull credentials to your laptop once (password is read over SSH, never passed on the command line):

esys-ci-host remote --host esysdocker image-registry credentials pull --save-as esysdocker

Then manage images on the VPS from your workstation:

# Docker engine cache on the VPS
esys-ci-host remote --host esysdocker image-registry cache list --filter libesys/
esys-ci-host remote --host esysdocker image-registry cache rm libesys/superbuild:ubuntu22.04

# Private registry at 127.0.0.1:5000
esys-ci-host remote --host esysdocker image-registry registry list
esys-ci-host remote --host esysdocker image-registry registry tags libesys/superbuild
esys-ci-host remote --host esysdocker image-registry registry push libesys/superbuild:ubuntu22.04
esys-ci-host remote --host esysdocker image-registry registry pull libesys/superbuild:ubuntu22.04
esys-ci-host remote --host esysdocker image-registry registry rm libesys/superbuild:ubuntu22.04

registry push runs docker login with --password-stdin on the VPS, then tags and pushes — equivalent to:

docker login 127.0.0.1:5000 -u libesys-ci --password-stdin
docker tag libesys/superbuild:ubuntu22.04 127.0.0.1:5000/libesys/superbuild:ubuntu22.04
docker push 127.0.0.1:5000/libesys/superbuild:ubuntu22.04

Credential resolution order: --password → ESYS_CI_HOST_IMAGE_REGISTRY_PASSWORD → saved host (credentials pull) → remote bootstrap.json over SSH.

Command	Purpose
image-registry prepare	Docker + localhost registry + daemon.json + gitlab-runner login
image-registry cache list	List images in the Docker engine cache
image-registry cache rm	Remove an image from the Docker cache
image-registry registry list	List repositories in the private registry
image-registry registry tags	List tags for one repository
image-registry registry push	Login, tag, and push a cached image to the registry
image-registry registry pull	Pull a registry image into the Docker cache
image-registry registry rm	Delete a tag from the registry
remote image-registry credentials pull	Save registry credentials under ~/.esys-ci-host/hosts/

Option	Purpose
--username NAME	htpasswd user (default: libesys-ci)
--password	Override generated registry password
--image-tag IMAGE	Override pinned registry:2 image
--skip-docker-install	Skip Docker apt install when Docker is already present

Local host registry

esys-ci-host remembers VPS hosts you configure on your workstation in ~/.esys-ci-host/hosts/<alias>.json (mode 600). Override the location with ESYS_CI_HOST_REGISTRY_DIR.

Two different "users" in the bootstrap command:

Flag	Meaning
remote --user root	Who you SSH as today on a fresh VPS (providers usually give root first)
host bootstrap --admin-user NAME	The new sudo account bootstrap creates (default: esysadmin)

After bootstrap with --save-host, follow-up remote --host artifactory reuses the saved admin user, IP, and identity file — you do not need to repeat --user or --identity-file unless you want to override them.

--save-host is only a local alias on your laptop — it does not configure Artifactory on the server. The name artifactory is just a label meaning "this VPS will be the registry host"; you could use --save-host contabo-vps instead. At bootstrap time the registry stores SSH details for the new admin user; Artifactory credentials are added later by artifactory deploy or hosts pull.

# First time only: bootstrap as root, save admin SSH details as alias "artifactory"
esys-ci-host remote --host 203.0.113.10 --user root --identity-file ~/.ssh/id_ed25519 \
  host bootstrap --yes --admin-user esysadmin --save-host artifactory --setup-ssh-config

# Later: --host artifactory is enough (user + key come from ~/.esys-ci-host/hosts/artifactory.json)
esys-ci-host remote --host artifactory host harden --yes
esys-ci-host remote --host artifactory --wheel dist/esys_ci_host-*.whl artifactory deploy --yes

esys-ci-host hosts list
esys-ci-host hosts show artifactory
esys-ci-host conan setup --host artifactory

After conan setup, upload packages with:

conan upload esystrace/0.1.0@libesys/stable -r libesys

Use saved aliases on remote — connection details are read from the registry when omitted:

esys-ci-host remote --host artifactory artifactory proxy --yes

Command	Purpose
artifactory deploy	Full flow: prepare + bootstrap + HTTPS (recommended)
artifactory prepare	Docker + Artifactory C++ CE on localhost ports
artifactory bootstrap	API: Conan repo, deploy user, saved credentials
artifactory proxy	Caddy + Let's Encrypt HTTPS for public hostname

Option	Purpose
--yes	Required to apply host changes
--dry-run	Print steps without mutating the host
--fail-step NAME	Force a controlled failure (testing)
--hostname HOST	Public hostname (default: artifactory.libesys.org)
--repo-name NAME	Conan repository key (default: libesys)
--admin-password	Override admin password on bootstrap re-run
--skip-verify	Skip HTTPS ping on proxy when DNS is not ready
--image-tag IMAGE	Override the pinned Artifactory C++ CE image
--skip-docker-install	Skip Docker apt install when Docker is already present

After bootstrap, configure Conan on your dev machine:

esys-ci-host conan setup --host artifactory
# or manually:
conan remote add libesys https://artifactory.libesys.org/artifactory/api/conan/libesys
conan remote login libesys libesys-conan -p '<deploy-password from bootstrap.json>'

Development

Architecture decisions are recorded in docs/adr/ (see INDEX.md).

Platform-specific host operations (packages, users, firewall, SSH) live under src/esys_ci_host/platform/. DebianAptBackend is implemented today; add new HostPlatformBackend subclasses (for example RhelDnfBackend) and register them in platform/registry.py to support additional distributions later. Runner install/prepare still use apt directly and will migrate to the platform layer in a follow-up.

python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
pre-commit install
pre-commit install --hook-type commit-msg
pytest

Release tags vX.Y.Z must match src/esys_ci_host/__version__.py. Never tag until master has a green pipeline on the commit you are releasing.

1. On develop: make bump-release (or python scripts/bump_release.py --yes --version X.Y.Z), then push and wait for green CI.
2. Fast-forward master from develop, push, and wait for green CI on master.
3. On master: make tag-release (or python scripts/tag_release.py --yes), then git push origin vX.Y.Z.

CI publishes tagged releases to GitLab PyPI and TestPyPI automatically; public PyPI is manual (publish:pypi).

Integration tests — fast suite mirrors GitLab CI; slow Artifactory test is local or manual CI only (tests/integration/README.md):

make integration-fast    # runner + bootstrap + remote-package + image-registry (~15–25 min)
make integration-slow    # artifactory prepare (30–60+ min); before release if artifactory changed

# Windows
.\test.ps1 -HostBootstrap
.\test.ps1 -Artifactory

When changing artifactory or remote prepare, run make integration-slow before tagging (or trigger manual integration:artifactory:ubuntu on GitLab).

License

Apache-2.0 — see LICENSE.