splent_feature_autodeploy

Release-gated continuous delivery for a SPLENT product. A product that selects this feature runs a small agent next to its web container in production. The agent follows a registry image channel, for example git.splent.io/splent-io/<product>:stable, and when the channel points to a new image it redeploys the product. The database is backed up first, the new version has to pass its health check, and an automatic rollback of both the container and the database when the new version does not come up.

The channel only moves when someone runs splent product:release, so nothing reaches production that nobody released. The agent never builds anything and never follows a branch.

How a check goes

Every AUTODEPLOY_INTERVAL seconds, and whenever an administrator asks, the agent goes through these steps.

1. Pull the channel image. If its id is the one the web container runs, there is nothing to do.
2. Skip an image that failed before, until the channel points elsewhere. Honour the pause flag and the update window (AUTODEPLOY_WINDOW). A check somebody asks for (Check now) deploys outside the window too.
3. Back up the database. mariadb-dump --single-transaction --routines --triggers runs as root inside the database container, with the credentials that container already holds, and the gzipped dump is kept only if mariadb-dump exited cleanly and the dump ends with its Dump completed trailer. No complete backup, no deploy.
4. Record what the web container runs (splent version --json).
5. Recreate the web container on the new image with everything it was configured with, the way Watchtower does it. That is its name, its environment (the container's own values on top of the new image's), its labels, mounts and binds, port bindings, restart policy, healthcheck, and its networks with their aliases. Cmd, Entrypoint and WorkingDir follow the new image when the old container only inherited them, and are kept when it chose them (the compose file's command:). The old container is renamed <name>-previous and stopped, not removed.
6. Wait for the healthcheck, up to AUTODEPLOY_HEALTH_TIMEOUT seconds. The entrypoint runs every feature's migrations before gunicorn starts, so this is generous.
7. Once it is healthy, remove the previous container and write the history entry (from and to image, product version, the features whose version changed, duration).
8. If it turns unhealthy, exits, restarts or runs out of time, keep the new container's last 80 log lines, remove it, drop and recreate the database and import the backup (so tables the new migrations created do not linger), put the previous container back under its name, wait for it to be healthy, and refuse that image from then on. The history entry says rolled_back and carries the log lines.

Every step is written to /data/state.json before it is taken. An agent restarted in the middle of a deploy (a reboot, a redeploy of the agent itself) finishes it if the new container turns healthy, and rolls it back otherwise.

Enable it in a product

1. Add the feature to the product line as an optional feature, from a detached workspace.

   splent spl:add-feature <spl> splent_feature_autodeploy
   

   In the UVL model it reads as below, with the constraint that keeps the admin panel it lives in.

   optional
       autodeploy {org 'splent-io', package 'splent_feature_autodeploy'}
   constraints
       autodeploy => admin
   

2. Attach it to the product and release the product, so the channel exists.

   splent product:select <product>
   splent feature:attach splent-io/splent_feature_autodeploy v0.1.0
   splent product:release
   

   A product release pushes <registry>/<org>/<product>:<version> and moves :stable to it.

3. On the server, set SPLENT_IMAGE in docker/.env.deploy to the channel the server follows, then build and deploy. The web service of the product's docker-compose.prod.yml must take its image from that variable, as in image: ${SPLENT_IMAGE:-<product>:latest}.

   SPLENT_IMAGE=git.splent.io/splent-io/<product>:stable
   

   splent product:build --no-image
   splent product:deploy
   

   product:build adds the agent's service to docker-compose.deploy.yml and copies docker/ to docker/features/splent_feature_autodeploy/. product:deploy offers a random value for AUTODEPLOY_SECRET, builds the agent's image and starts it.

4. For a private registry set AUTODEPLOY_REGISTRY_USER and AUTODEPLOY_REGISTRY_TOKEN (a token that can read packages).

From then on, splent product:release is the whole deploy.

Configuration

Declared in [tool.splent.config], so they appear in .env.deploy.example with the product's names filled in.

The Deployments page

/admin/deployments, in the admin panel's sidebar, for administrators only (the admin feature's ADMIN_ROLES). It shows the channel, the running image and product version, what the agent is doing and why, the last check, the update window, whether updates are paused, and the latest deploys with the features each one moved. A rolled back deploy shows its log lines in a collapsible block, and a banner says so until the next deploy. Check now, Pause and Resume are CSRF protected forms.

When the agent is missing (development), stopped or refuses the secret the page explains what is going on. The site itself is never affected.

Commands

splent feature:autodeploy status [--json]
splent feature:autodeploy history [--limit N] [--json]
splent feature:autodeploy check
splent feature:autodeploy pause
splent feature:autodeploy resume

They talk to the agent's API, so they need the product network. On a production server run them in the web container.

docker exec <product>_web_deploy splent feature:autodeploy status

The agent's API

Port 8080 on the product network, never published on the host. Every request carries Authorization: Bearer <AUTODEPLOY_SECRET>, compared in constant time; anything else is a 401, whatever the path.

There is no rollback endpoint on purpose. A rollback restores the backup taken before the deploy, so run later by hand it would throw away everything written since.

Security

• The agent mounts /var/run/docker.sock. Whoever controls that socket is root on the host, so the agent is as sensitive as the host itself. It runs only in production, from the image product:deploy builds out of this feature's docker/ directory, and does only what this README describes.
• It publishes no port. Only containers on the product network reach the API, and every call needs the shared secret. The secret never appears in the status, the history or the logs.
• The web side calls the agent without any proxy, so an HTTP_PROXY in the environment never sees the secret.
• The database credentials stay in the database container. The dump and the restore read them from that container's environment and pass them through MYSQL_PWD, so they are not on any command line and are never printed.
• Registry credentials are only sent to the registry, with the pull.

Backups

One per deploy, in the agent's autodeploy_data volume under /data/backups/, named <UTC time>-<short id of the image being replaced>.sql.gz, so each file says which code its schema belongs to. The newest AUTODEPLOY_KEEP_BACKUPS are kept. They live on the same host as the database, so they protect against a bad release, not against losing the server. Copy them elsewhere if you need that.

Restoring a backup by hand

Restoring a backup loses everything written after it was taken. Pause the agent first, so it does not deploy in the middle.

docker exec <product>_web_deploy splent feature:autodeploy pause
docker cp <product>_autodeploy:/data/backups/<file>.sql.gz .
docker stop <product>_web_deploy
docker exec <product>_db_deploy bash -c 'MYSQL_PWD="$MARIADB_ROOT_PASSWORD" mariadb -uroot -e "DROP DATABASE IF EXISTS \`$MARIADB_DATABASE\`; CREATE DATABASE \`$MARIADB_DATABASE\`"'
gunzip -c <file>.sql.gz | docker exec -i <product>_db_deploy bash -c 'MYSQL_PWD="$MARIADB_ROOT_PASSWORD" mariadb -uroot "$MARIADB_DATABASE"'

Then run a web container on the image named in the file name, so code and schema match, and resume the agent.

To let the agent try an image it refused, pause it, delete the image id from bad_images in /data/state.json in the agent's volume, and resume.

What the agent does not do

• It replaces the web container only. A release that adds a service (a feature that ships its own container), or needs a new setting without a default in .env.deploy, has to go through splent product:deploy; pause the agent while you do it. Left to the agent such a release fails its health check and is rolled back, which is safe but not what you want.
• It does not update itself. A new version of this feature reaches the server with splent product:deploy, which rebuilds the agent's image.
• It replaces one web container, not a set of replicas.

Develop

The agent is plain Python 3.13 with docker-py, in docker/autodeploy_agent/. It gets the Docker client handed in, so its tests run against a fake Engine.

splent feature:test splent_feature_autodeploy

From a bare checkout, with a Python that has the framework, Flask-Babel, Flask-Login and Flask-WTF installed, the same tests run without a product.

python -m pytest src/splent_io/splent_feature_autodeploy/tests