splent_feature_autodeploy

Release-gated continuous delivery for a SPLENT product. A product that selects this feature runs a small agent in a container of its own next to its web container in production. The agent follows a registry image channel, for example git.splent.io/splent-io/<product>:stable, and when the channel points to a new image it redeploys the product on that image's digest. It backs up first (off the host too, when the product has splent_feature_backup), migrates once, waits for the new version's /readyz, and rolls the container back, and the database when the deploy changed it, when the new version is not ready.

The channel only moves when someone runs splent product:release, so nothing reaches production that nobody released. The agent never builds anything and never follows a branch.

How a check goes

Every AUTODEPLOY_INTERVAL seconds, and whenever an administrator asks, the agent goes through these steps.

1. Pull the channel image. If its id is the one the web container runs, there is nothing to do. Otherwise the image is named from now on by its digest (<registry>/<org>/<product>@sha256:...), which no push can move: that is what the new container runs and what the history records. An image the registry knows no digest of runs by its id, just as fixed.

2. Skip an image that failed before, until the channel points elsewhere. Honour the pause flag and the update window (AUTODEPLOY_WINDOW). A check somebody asks for (Check now) deploys outside the window too.

3. Back up.

   • When the product has splent_feature_backup, run its off-host backup in the running web container, as the app's user: splent feature:backup run --tag pre-deploy-<short id of the image being replaced> (AUTODEPLOY_BACKUP_COMMAND). It is encrypted to keys the server does not hold, and it is the copy that survives the host.
   • Then the agent's own dump, the one a rollback restores: mariadb-dump --single-transaction --routines --triggers runs as root inside the database container, with the credentials that container already holds, and the gzipped dump is kept only if mariadb-dump exited cleanly and the dump ends with its Dump completed trailer.

   No complete backup, no deploy.

4. Record what the web container runs (splent version --json), and a fingerprint of the schema: every table's columns and every migration revision (alembic_*, splent_migrations).

5. Create the new web container on the image's digest with everything the old one was configured with, the way Watchtower does it. That is its name, its environment (the container's own values on top of the new image's), its labels, mounts and binds, port bindings, restart policy, healthcheck, and its networks with their aliases. Cmd, Entrypoint and WorkingDir follow the new image when the old container only inherited them, and are kept when it chose them (the compose file's command:). The old container is renamed <name>-previous and stopped, not removed.

6. Migrate, once: a one-off container of the new image, <name>-migrate, with the web container's environment, mounts and networks, runs AUTODEPLOY_MIGRATE_COMMAND (splent db:upgrade) as the app's user while neither version serves. It answers to no alias, so nginx never finds it, publishes no port and is removed once it exits. The new container's entrypoint upgrades again as it starts and finds nothing left to do.

7. Start the new container and wait, up to AUTODEPLOY_HEALTH_TIMEOUT seconds, for /readyz to answer 200 (AUTODEPLOY_READY_PATH, asked by the container's name on AUTODEPLOY_WEB_PORT), while Docker's healthcheck does not say unhealthy. /readyz is the framework's (1.22.0 and later): 200 when the database and the services the product declares answer. An image without it answers 404, and the healthcheck decides for it.

8. Once it is ready, remove the previous container and write the history entry (from and to image, the digest, product version, the features whose version changed, the off-host backup, duration).

9. If the migration fails, or the new container turns unhealthy, exits, restarts or is not ready in time, keep its last 80 log lines (the migration's, when that is what failed), remove it, put the database back when the rule below says so, put the previous container back under its name, wait for it to be healthy, and refuse that image from then on. The history entry says rolled_back, carries the log lines and says whether the database went back and why.

When the database goes back

A rollback restores the agent's pre-deploy dump (dropping the database first, so tables the new migrations created do not linger) when:

• the migration failed, or the agent died during it. MariaDB commits DDL as it goes, so a failed migration can leave half of its changes behind without recording them;
• the migration finished, and the schema fingerprint differs from the one taken before the deploy. The previous version cannot run on the new schema, and a down-migration is not something to trust with production data;
• the fingerprint cannot be read, which counts as changed;
• AUTODEPLOY_MIGRATE_COMMAND=off: the new container migrated as it started, and the agent cannot tell a failed migration from anything else.

It does not when the deploy stopped before anything ran against the database (a failed backup, a container that could not be created), and when the migration finished without changing the schema: the previous version comes back on the same database, and nothing written while the new one tried to start is lost. The dump is taken right before the old version stops, so a restore loses only what the new version wrote during its attempt.

Every step is written to /data/state.json before it is taken. An agent restarted in the middle of a deploy (a reboot, a redeploy of the agent itself) finishes it if the new container turns ready, and rolls it back otherwise; one restarted during the migration rolls back with the database.

State	Meaning
idle	The web container runs the image the channel points to
waiting_for_window	A new image is ready and waits for the update window
deploying	Backup, recreate or health wait in progress
rolling_back	The new image failed and the previous version is coming back
paused	An administrator paused automatic deploys
held	The only image available failed before; waiting for the next release
error	Something needs a person, the detail says what
not_configured	A required setting is missing, the detail lists it

Enable it in a product

1. Add the feature to the product line as an optional feature, from a detached workspace.

   splent spl:add-feature <spl> splent_feature_autodeploy
   

   In the UVL model it reads as below, with the constraint that keeps the admin panel it lives in.

   optional
       autodeploy {org 'splent-io', package 'splent_feature_autodeploy'}
   constraints
       autodeploy => admin
   

2. Attach it to the product and release the product, so the channel exists.

   splent product:select <product>
   splent feature:attach splent-io/splent_feature_autodeploy v0.1.0
   splent product:release
   

   A product release pushes <registry>/<org>/<product>:<version> and moves :stable to it.

3. On the server, set SPLENT_IMAGE in docker/.env.deploy to the channel the server follows, then build and deploy. The web service of the product's docker-compose.prod.yml must take its image from that variable, as in image: ${SPLENT_IMAGE:-<product>:latest}.

   SPLENT_IMAGE=git.splent.io/splent-io/<product>:stable
   

   splent product:build --no-image
   splent product:deploy
   

   product:build adds the agent's service to docker-compose.deploy.yml and copies docker/ to docker/features/splent_feature_autodeploy/. product:deploy offers a random value for AUTODEPLOY_SECRET, builds the agent's image and starts it.

4. For a private registry set AUTODEPLOY_REGISTRY_USER and AUTODEPLOY_REGISTRY_TOKEN (a token that can read packages).

From then on, splent product:release is the whole deploy.

Configuration

Declared in [tool.splent.config], so they appear in .env.deploy.example with the product's names filled in.

Variable	Default	Meaning
AUTODEPLOY_IMAGE	empty	Image to follow. Empty means SPLENT_IMAGE. A channel (:stable) or a version tag; :latest, or no tag at all, is refused
AUTODEPLOY_INTERVAL	300	Seconds between checks (at least 30)
AUTODEPLOY_WINDOW	empty	UTC span for deploys, like 02:00-05:00 or 22:00-04:00. Empty means any time
AUTODEPLOY_HEALTH_TIMEOUT	600	Seconds the migrations get, and then the new container to turn ready
AUTODEPLOY_MIGRATE_COMMAND	splent db:upgrade	What the one-off migration container runs. off leaves the migrations to the new container's start
AUTODEPLOY_READY_PATH	/readyz	The readiness probe. off waits for Docker's healthcheck alone
AUTODEPLOY_WEB_PORT	5000	The port the web container answers on, inside the product network
AUTODEPLOY_BACKUP_COMMAND	auto	The off-host backup before a deploy. auto is splent_feature_backup's when the product has it, off none, anything else the command to run in the web container
AUTODEPLOY_APP_USER	splent	The user the migration and the backup command run as
AUTODEPLOY_KEEP_BACKUPS	5	Rollback dumps kept, newest first
AUTODEPLOY_WEB_CONTAINER	<product>_web_deploy	The container that is replaced
AUTODEPLOY_DB_CONTAINER	<product>_db_deploy	The MariaDB container that is backed up
AUTODEPLOY_AGENT_CONTAINER	<product>_autodeploy	The agent's container name
AUTODEPLOY_URL	http://<product>_autodeploy:8080	Where the web side finds the agent
AUTODEPLOY_REGISTRY_USER	empty	Registry account, for a private channel
AUTODEPLOY_REGISTRY_TOKEN	empty	Registry token, for a private channel
AUTODEPLOY_SECRET	<SET>	Shared by the web container and the agent

The Deployments page

/admin/deployments, in the admin panel's sidebar, for administrators only (the admin feature's ADMIN_ROLES). It shows the channel, the running image and product version, what the agent is doing and why, the last check, the update window, whether updates are paused, and the latest deploys with the features each one moved. A rolled back deploy shows its log lines in a collapsible block, and a banner says so until the next deploy. Check now, Pause and Resume are CSRF protected forms.

When the agent is missing (development), stopped or refuses the secret the page explains what is going on. The site itself is never affected.

Commands

splent feature:autodeploy status [--json]
splent feature:autodeploy history [--limit N] [--json]
splent feature:autodeploy check
splent feature:autodeploy pause
splent feature:autodeploy resume

They talk to the agent's API, so they need the product network. On a production server run them in the web container.

docker exec <product>_web_deploy splent feature:autodeploy status

The agent's API

Port 8080 on the product network, never published on the host. Every request carries Authorization: Bearer <AUTODEPLOY_SECRET>, compared in constant time; anything else is a 401, whatever the path.

Method	Path	Does
GET	/status	State, configuration, running versions, backups
GET	/history?limit=N	The newest N deploys, newest first (1 to 200)
POST	/check	Look for a new image now
POST	/pause	Stop deploying; a deploy in progress finishes
POST	/resume	Deploy again, and check right away

There is no rollback endpoint on purpose. A rollback restores the backup taken before the deploy, so run later by hand it would throw away everything written since.

Security

• The agent mounts /var/run/docker.sock. Whoever controls that socket is root on the host, so the agent is as sensitive as the host itself. It runs only in production, in a container of its own, from the image product:deploy builds out of this feature's docker/ directory, and does only what this README describes. No container of the app mounts the socket: not the web container, not the migration job it creates.
• It publishes no port. Only containers on the product network reach the API, and every call needs the shared secret. The secret never appears in the status, the history or the logs.
• The web side calls the agent without any proxy, so an HTTP_PROXY in the environment never sees the secret.
• The database credentials stay in the database container. The dump and the restore read them from that container's environment and pass them through MYSQL_PWD, so they are not on any command line and are never printed.
• Registry credentials are only sent to the registry, with the pull.

Backups

One per deploy, in the agent's autodeploy_data volume under /data/backups/, named <UTC time>-<short id of the image being replaced>.sql.gz, so each file says which code its schema belongs to. The newest AUTODEPLOY_KEEP_BACKUPS are kept. They live on the same host as the database, so they protect against a bad release, not against losing the server: that is what splent_feature_backup's off-host backup, taken just before, is for. Without that feature, copy them elsewhere if you need it.

Restoring a backup by hand

Restoring a backup loses everything written after it was taken. Pause the agent first, so it does not deploy in the middle.

docker exec <product>_web_deploy splent feature:autodeploy pause
docker cp <product>_autodeploy:/data/backups/<file>.sql.gz .
docker stop <product>_web_deploy
docker exec <product>_db_deploy bash -c 'MYSQL_PWD="$MARIADB_ROOT_PASSWORD" mariadb -uroot -e "DROP DATABASE IF EXISTS \`$MARIADB_DATABASE\`; CREATE DATABASE \`$MARIADB_DATABASE\`"'
gunzip -c <file>.sql.gz | docker exec -i <product>_db_deploy bash -c 'MYSQL_PWD="$MARIADB_ROOT_PASSWORD" mariadb -uroot "$MARIADB_DATABASE"'

Then run a web container on the image named in the file name, so code and schema match, and resume the agent.

To let the agent try an image it refused, pause it, delete the image id from bad_images in /data/state.json in the agent's volume, and resume.

What the agent does not do

• It replaces the web container only. A release that adds a service (a feature that ships its own container), or needs a new setting without a default in .env.deploy, has to go through splent product:deploy; pause the agent while you do it. Left to the agent such a release fails its readiness check and is rolled back, which is safe but not what you want.
• It takes no database lock (GET_LOCK) around the migration: one agent replaces one web container, which is stopped while the job runs, so the job is the only migrator. Several replicas would need that lock.
• It does not update itself. A new version of this feature reaches the server with splent product:deploy, which rebuilds the agent's image.
• It replaces one web container, not a set of replicas.

Develop

The agent is plain Python 3.13 with docker-py, in docker/autodeploy_agent/. It gets the Docker client handed in, so its tests run against a fake Engine.

splent feature:test splent_feature_autodeploy

From a bare checkout, with a Python that has the framework, Flask-Babel, Flask-Login and Flask-WTF installed, the same tests run without a product.

python -m pytest src/splent_io/splent_feature_autodeploy/tests