swim-protocol 0.3.12

SWIM protocol implementation for exchanging cluster membership status and metadata.

swim-protocol

SWIM protocol implementation for exchanging cluster membership status and metadata.

This library is intended to fit into an asyncio event loop to help synchronize a distributed group of processes.

Introduction

API Documentation

Install and Usage

$ pip install swim-protocol

Running the Demo

There is a demo application included as a reference implementation. Try it out by running the following, each from a new terminal window, and use Ctrl-C to exit:

$ swim-protocol-demo -c --name 127.0.0.1:2001 --peer 127.0.0.1:2003 --metadata name one
$ swim-protocol-demo -c --name 127.0.0.1:2002 --peer 127.0.0.1:2001 --metadata name two
$ swim-protocol-demo -c --name 127.0.0.1:2003 --peer 127.0.0.1:2001 --metadata name three
$ swim-protocol-demo -c --name 127.0.0.1:2004 --peer 127.0.0.1:2003 --metadata name four

Typing in any window will disseminate what has been typed across the cluster with eventual consistency.

Getting Started

First you should create a new UdpConfig object:

from swimprotocol.udp import UdpConfig

config = UdpConfig(local_name='127.0.0.1:2001',
                   local_metadata={'name': b'one'},
                   peers=['127.0.0.1:2002'])

All other config arguments have default values, which are tuned somewhat arbitrarily with a small cluster of 3-4 members in mind.

Now you can create the cluster members manager and transport layer, and enter the event loop:

from contextlib import AsyncExitStack
from swimprotocol.members import Members
from swimprotocol.udp import UdpTransport

transport = UdpTransport(config)
members = Members(config)
async with AsyncExitStack() as stack:
    worker = await stack.enter_async_context(transport.enter(members))
    await worker.run()  # or schedule as a task

These snippets demonstrate the UDP transport layer directly. For a more generic approach that uses argparse and load_transport, check out the demo.

If your application is deployed as a Docker Service, the UdpConfig discovery=True keyword argument can be used to discover configuration based on the service name. See the documentation for more comprehensive usage.

Checking Members

The Members object provides a few ways to check on the cluster and its members:

for member in members.non_local:
    # all other known cluster members
    print(member.name, member.status, member.metadata)

from swimprotocol.status import Status
for member in members.get_status(Status.AVAILABLE):
    # all cluster members except offline
    print(member.name, member.status, member.metadata)

Alternatively, listen for status or metadata changes on all members:

from swimprotocol.member import Member

async def _updated(member: Member) -> None:
    print('updated:', member.name, member.status, member.metadata)

async with AsyncExitStack() as stack:
    # ...
    stack.enter_context(members.listener.on_notify(_updated))

UDP Transport Security

The UdpTransport transport layer (the only included transport implementation) uses salted hmac digests to sign each UDP packet payload. Any UDP packets received that are malformed or have an invalid signature are silently ignored. The eventual consistency model should recover from packet loss.

The signatures rely on a shared secret between all cluster members, given as the secret=b'...' argument to the Config constructor. If secret=None is used, it defaults to uuid.getnode() but this is not secure for production setups unless all sockets are bound to a local loopback interface.

The cluster member metadata is not encrypted during transmission, so only private networks should be used if metadata includes any secret data, or that secret data should be encrypted separately by the application. Also be aware that low MTU sizes on public networks may affect the ability to synchronize larger amounts of metadata.

Development

You will need to do some additional setup to develop and test plugins. Install Hatch to use the CLI examples below.

Run all tests and linters:

$ hatch run check

Because this project supports several versions of Python, you can use the following to run the checks on all versions:

$ hatch run all:check

Type Hinting

This project makes heavy use of Python's type hinting system, with the intention of a clean run of mypy:

$ mypy

No code contribution will be accepted unless it makes every effort to use type hinting to the extent possible and common in the rest of the codebase.