lfp-swarm-consul

lfp-swarm-consul runs Consul servers on eligible Docker Swarm managers and continuously reconciles their membership. Every manager also runs a controller contender. The bundled lfp-swarm-lease run supervisor starts the Python reconciler only on the current lease holder and stops its complete process tree when lease renewal fails.

Consul still elects its own Raft leader. The Swarm lease only elects the controller authorized to change deployment and membership state.

What is implemented

• A global Consul server service with manager-local persistent data.
• Gossip encryption supplied by an external Swarm secret.
• Automatic initial bootstrap for a configured server count, including two-node labs.
• Removal of bootstrap_expect after Consul has a leader and the expected peer set.
• Continuous Swarm manager, service task, Consul LAN, Raft, and Autopilot observation.
• One-at-a-time joins for running tasks missing from Consul membership.
• A separate zero-replica Swarm service containing fence-aware controller state and operation intent.
• Crash recovery through persisted intent and observed postconditions.
• Explicit recovery-required state when Raft peers exist but no Consul leader can be found.
• Optional forced removal after Swarm proves a server is no longer eligible and Autopilot is healthy.

Forced removal is disabled by default. An unreachable manager remains desired; reachability alone never authorizes removal.

Runtime

controller task on every manager
            │
            └── lfp-swarm-lease election
                       │
                       └── active fence starts Python reconciler
                                      │
              ┌───────────────────────┴──────────────────────┐
              │                                              │
       Docker Swarm API                               Consul HTTP API
       managers + tasks                         members + Raft + Autopilot
              │                                              │
              └──────── one persisted action at a time ──────┘

The controller state is stored in consul_controller-state. Lease state remains owned separately by lfp-swarm-lease.

Build the controller image

The image installs both the reconciler and the pure-Python lease supervisor from locked uv dependencies. lfp-swarm-lease is resolved from PyPI, so release and container builds do not depend on an adjacent source checkout:

docker buildx build \
  --tag registry.example.com/lfp-swarm-consul:0.1.0 \
  --push \
  .

Python dependencies are locked in uv.lock. Local verification uses:

uv sync --locked
uv run ruff check .
uv run ruff format --check .
uv run pytest

Tagged releases use the same release GitHub environment and PyPI trusted-publishing model as lfp-swarm-lease. The tag must match uv version --short; for example, run uv version 0.1.1, commit the updated project metadata and lockfile, then push v0.1.1.

The supervisor's observation API remains available on port 8080. It starts lfp-swarm-consul only after acquiring a lease and supplies the current term through SWARM_LEASE_FENCE. A lost lease stops the controller process group; the same holder is not restarted until ownership first transitions away and a later acquisition produces a new fence.

Deploy the stack

The supplied stack must be deployed with the name consul; its internal service names are consul_server and consul_controller.

Create the gossip key without placing its value in the repository or command arguments:

docker run --rm \
  hashicorp/consul:1.22.7@sha256:a230dcea0bb107bd7958a912d1429fb7f9d399637de7ffb814b34412b9e8c543 \
  keygen |
  docker secret create consul-gossip-key -

Render the exact deployment input before deploying:

export CONSUL_CONTROLLER_IMAGE=registry.example.com/lfp-swarm-consul:0.1.0
export CONSUL_BOOTSTRAP_EXPECT=2
docker stack config --compose-file stack/lfp-swarm-consul.stack.yml
docker stack deploy --compose-file stack/lfp-swarm-consul.stack.yml consul

Swarm does not build images during docker stack deploy; publish the controller image first.

LFPConnect demo deployment

lfp-swarm-consul.lfpconnect.stack.yml is the Portainer-owned deployment for the isolated LFPConnect demo. It runs one Consul server and one controller contender on each of the three Swarm managers. Traefik load-balances the controllers' read-only lease observation API at https://consul-demo.lfpconnect.io; the Consul HTTP API remains private to the stack network.

The deployment expects these external Swarm objects, created without generation suffixes because they are the first generation:

• lfp-swarm-consul-server-config
• lfp-swarm-consul-server-entrypoint-v3 (earlier configs remain immutable)
• lfp-swarm-consul-gossip-key

The checked-in LFPConnect definition records the live cluster's post-bootstrap steady state, so the server service deliberately omits CONSUL_BOOTSTRAP_EXPECT. To recreate the cluster from empty volumes, temporarily set it to 3 for the first deployment. The controller removes it after the initial peer set is healthy; return the Portainer definition to the checked-in steady state after that transition. When a new manager is added later, the global service starts a server there and retry_join plus the controller admit it without changing the original bootstrap count.

Two-node behavior

Two servers are supported with CONSUL_BOOTSTRAP_EXPECT=2, but the topology has no failure tolerance: both voters are required for quorum. The controller reports that condition and never automatically reboots a new generation after quorum loss. Removing bootstrap_expect performs a one-at-a-time rolling restart and therefore causes a brief write outage in a two-voter cluster.

A third manager and Consul server is the recommended resilient topology.

Eligibility and removal

Every active Swarm manager is eligible by default. Set this node label before draining a server when it should no longer host Consul:

docker node update --label-add lfp.consul.server=false <node>

The stack placement constraint and planner both honor that label. The server service uses leave_on_terminate, so a reachable task normally leaves gracefully. Stale forced removal remains disabled unless CONSUL_ALLOW_FORCE_REMOVE=true is set for a deployment. Enable it only when the Swarm membership is authoritative and recovery procedures are understood.

Security boundary

The control network is an encrypted overlay and gossip traffic has a Swarm-secret key. The Consul HTTP API is not published outside that network. ACL bootstrap, HTTPS certificates, and a least-privilege controller token are intentionally deployment-specific and are not fabricated by this stack. Configure those before exposing the API or joining untrusted workloads.

Mounting /var/run/docker.sock gives controller containers manager-equivalent authority. Only the controller service receives that mount.

See docs/PLAN.md for reconciliation invariants and recovery boundaries.

The live three-manager failure exercise, including node removal, controller and leader loss, a simultaneous six-container crash, and KV integrity verification, is recorded in docs/FAILURE-RECOVERY-2026-10-15.md.