lfp-swarm-lease

lfp-swarm-lease elects one leader for one Docker Swarm service. It stores the election in Swarm's Raft-backed service metadata, so it needs no database and no fixed manager.

The implementation is pure Python. The controller, CLI, FastAPI server, and Python observer ship as one Python package built with uv and Hatchling. The TypeScript observer is generated from the controller's OpenAPI contract and contains no native binaries.

pip install lfp-swarm-lease
npm install @regbo/lfp-swarm-lease

The model is intentionally narrow:

• One deployed controller service equals one election.
• The namespace defaults to the current Swarm service name.
• The holder ID defaults to the current Swarm node ID.
• Acquisition is local-only and requires Docker manager access.
• Remote clients are observation-only and use HTTP/SSE.
• A different independent election should be a different controller service.

Run a leader-only process

Use the CLI as every manager replica's entrypoint:

lfp-swarm-lease run -- ./consul-controller

The process keeps contending in the background. Each time this controller becomes leader it starts the command with SWARM_LEASE_FENCE, SWARM_LEASE_HOLDER_ID, SWARM_LEASE_NODE_ID, SWARM_LEASE_TASK_ID, and SWARM_LEASE_NAMESPACE. If renewal fails or the fence changes, it sends SIGTERM to the child process group, waits 10 seconds, then kills remaining children. A later transition back to this node starts a fresh process. --once exits after the first leadership term.

Every running contender also serves:

Endpoint	Purpose
GET /healthz	Process liveness without a Docker call.
GET /readyz	Docker/Swarm readiness.
GET /v1/info	Bound namespace.
GET /v1/state	Complete current election snapshot.
GET /v1/events	SSE current holder, then holder/fence changes.
GET /docs	Interactive OpenAPI documentation.

The first SSE event is current. Later events are change. Routine renewals retain the fence and are suppressed. Clients can connect to one Swarm service hostname: the VIP may round-robin reconnections because every replica observes the same anchor.

acquired has two contexts. In TryAcquireResult it means that acquisition attempt won. In snapshots and watcher events it means the election currently has some holder; compare holder_id, node_id, or task_id when caller-relative ownership is needed.

Discovery and configuration

Configuration precedence is:

1. Typed CLI or ControllerOptions values.
2. Environment variables.
3. Current Swarm task labels.
4. Defaults.

By default all variables begin with SWARM_LEASE_. --env-prefix POSTGRES switches the complete set to POSTGRES_*.

For automatic discovery, mount the Docker socket and leave the container hostname at Docker's default:

volumes:
  - /var/run/docker.sock:/var/run/docker.sock

The controller inspects the container named by HOSTNAME and reads:

• com.docker.swarm.service.name as the namespace
• com.docker.swarm.node.id as node metadata and the default holder ID
• com.docker.swarm.task.id as task metadata

The hostname is only the lookup key. If it does not resolve to a container with a Swarm task ID and the namespace is still needed, startup fails immediately. Set SWARM_LEASE_NAMESPACE and SWARM_LEASE_HOLDER_ID explicitly when using a hard-coded hostname.

Important settings include:

• SWARM_LEASE_TTL_MS — lease duration, default 30 seconds.
• SWARM_LEASE_MAX_CAS_RETRIES — collisions retried within one mutation, default 8. It never limits how long run contends.
• SWARM_LEASE_LISTEN — HTTP bind address, default 0.0.0.0:8080.
• SWARM_LEASE_DEFAULT_POLL_MS — watcher poll cadence, default 1 second.
• SWARM_LEASE_KEEP_ALIVE_MS — SSE keepalive, default 15 seconds.
• SWARM_LEASE_SHUTDOWN_TIMEOUT_MS — child grace period, default 10 seconds.

init is optional because acquisition creates the zero-replica anchor automatically.

Python controller API

The Python controller is async and typed with Pydantic:

import asyncio

from lfp_swarm_lease import ControllerOptions, SwarmLeaseController

async def main() -> None:
    controller = await SwarmLeaseController.connect(
        ControllerOptions(namespace="consul-controllers", holder_id="manager-a")
    )
    result = await controller.try_acquire()
    if result.grant is None:
        print("held by", result.current_holder_id)
        return

    grant = result.grant
    await consul.reconcile_leader(fencing_token=grant.fence)
    grant = await controller.renew(grant)
    await controller.release(grant)

asyncio.run(main())

In Swarm, omit the two explicit values and let task discovery fill them. try_acquire() is nonblocking. run is usually safer than hand-writing the renewal and process-lifecycle loop.

Watch from Python

import asyncio

from lfp_swarm_lease import RemoteLeaseObserver

async def main() -> None:
    watcher = RemoteLeaseObserver.connect(
        "http://controllers:8080"
    ).watch()
    async for state in watcher:
        await status.publish(
            acquired=state.acquired,
            holder_id=state.holder_id,
            fence=state.fence,
        )

asyncio.run(main())

The observer waits up to 60 seconds for an initial controller response. After the first state it reconnects indefinitely with jittered exponential backoff. Replayed state after a controller restart is suppressed. wait_for_holder(timeout) skips inactive states.

Watch from TypeScript

import { RemoteLeaseObserver } from "@regbo/lfp-swarm-lease";

const watcher = RemoteLeaseObserver.connect(
  "http://controllers:8080",
).watch();

for await (const state of watcher) {
  await status.publish({
    acquired: state.acquired,
    holderId: state.holder_id,
    fence: state.fence,
  });
}

createLeaseClient() exposes the ordinary OpenAPI-typed endpoints. The SSE wrapper adds transparent retry, timeout, and transition de-duplication.

PostgreSQL or Consul controller stub

The election should run the coordinator, not the database server itself:

exec lfp-swarm-lease run -- ./postgres-controller

# postgres-controller
fence = int(os.environ["SWARM_LEASE_FENCE"])
postgres.promote_if_safe(fencing_token=fence)
postgres.serve_until_sigterm()
postgres.demote_and_fence_writes()

For Consul, the child can reconcile catalog, ACL, configuration, or an external resource while it is leader. Consul server membership and its own Raft leader election remain Consul's responsibility. This lease chooses one orchestration process; it does not replace Consul consensus.

The fencing token matters even when holder_id stays the same: a restarted controller on the same manager gets a newer fence. A protected resource that can reject stale tokens provides the strongest split-brain protection.

Other CLI commands

lfp-swarm-lease init
lfp-swarm-lease try-acquire
lfp-swarm-lease renew 7
lfp-swarm-lease release 7
lfp-swarm-lease inspect
lfp-swarm-lease destroy

try-acquire exits 75 when another contender holds the election. release exits 3 when the identity/fence pair does not match.

Development

uv sync --frozen
uv run ruff check .
uv run mypy src
uv run pytest
uv build --no-sources

bun install --frozen-lockfile
bun run node:test
bun run demo:test
bun run demo:typecheck
bun run demo:lint
bun run demo:build

The default uv build flow uses the Hatchling source-build hook to export the FastAPI contract and generate the TypeScript client before packaging Python. Run bun run clients:build directly while iterating only on the TypeScript client.

Release and demo deployment

Python, the generated Node client, and the private demo carry the same version. The local release command defaults to a patch bump and also accepts minor or major:

bun run release --dry-run
bun run release          # 0.1.0 -> 0.1.1
bun run release minor    # 0.1.1 -> 0.2.0
bun run release major    # 0.2.0 -> 1.0.0

It requires a clean branch that is not behind its upstream, updates both lockfiles, runs the release tests, commits the version files, creates an annotated vX.Y.Z tag, and atomically pushes the commit and tag. The tag starts the release workflow, which validates the shared version, publishes through PyPI trusted publishing and the configured npm automation token, then publishes the controller and demo images to Docker Hub as X.Y.Z and latest. Release CI never receives Portainer credentials and never changes a running stack.

Repository variables BUILD_ARM=false and PUBLISH_NPM=false disable those optional release steps. A manual workflow run exposes the same switches. All three artifact types always use the version validated from the tag or manual version input.

Demo lifecycle is an explicit local command. It supports local/private image pushes and Portainer deployment without reimplementing package publishing:

bun run demo --help
bun run demo deploy --dry-run
bun run demo deploy
bun run demo remove  # `rm` is an alias

Configuration precedence is CLI, process environment, process *_FILE, .env, .env *_FILE, then Pydantic defaults. Copy .env.example to the ignored .env for a local setup. Every setting uses the SWARM_LEASE_DEPLOY_ prefix, and every setting accepts a matching _FILE form so tokens never need to be placed directly in the environment or on a command line.

demo deploy pushes both images to Docker Hub or a private registry, resolves the demo image digest, renders and validates deploy/stack.yml, and only then creates or updates the exact Portainer-owned stack. --no-build-arm limits local image builds to linux/amd64. --no-push-images --image-tag VERSION deploys an image that the release workflow already published. demo remove is idempotent and verifies the configured stack name and endpoint before deleting it; it leaves volumes and unrelated stacks alone.

The release workflow consumes publishing values only from GitHub variables and secrets. Set repository variables BUILD_ARM and PUBLISH_NPM for tag-triggered defaults. Configure the release environment with:

• Variable: CONTAINER_NAMESPACE.
• Secrets: CONTAINER_REGISTRY_USERNAME, CONTAINER_REGISTRY_TOKEN, and NPM_TOKEN.

PyPI uses trusted publishing, so the GitHub release environment must be added as a trusted publisher for this repository on PyPI. NPM_TOKEN must be a granular automation token permitted to publish without an interactive 2FA prompt.

Safety

• A partition without Swarm write quorum cannot acquire or renew.
• The Docker socket is manager-equivalent access and belongs only on trusted controllers.
• Observation clients never receive Docker access.
• Expiration uses wall-clock time; keep manager clocks synchronized.
• Authentication, when needed, belongs in HTTP middleware or a reverse proxy.