swim-protocol 0.3.4

SWIM protocol implementation for exchanging cluster membership status and metadata.

swim-protocol

SWIM protocol implementation for exchanging cluster membership status and metadata.

This library is intended to fit into an asyncio event loop to help synchronize a distributed group of processes.

Introduction

API Documentation

Install and Usage

$ pip install swim-protocol

Running the Demo

There is a demo application included as a reference implementation. Try it out by running the following, each from a new terminal window, and use Ctrl-C to exit:

$ swim-protocol-demo -c --name 127.0.0.1:2001 --peer 127.0.0.1:2003 --metadata name one
$ swim-protocol-demo -c --name 127.0.0.1:2002 --peer 127.0.0.1:2001 --metadata name two
$ swim-protocol-demo -c --name 127.0.0.1:2003 --peer 127.0.0.1:2001 --metadata name three
$ swim-protocol-demo -c --name 127.0.0.1:2004 --peer 127.0.0.1:2003 --metadata name four

Every 10 seconds or so, each member will randomize its token metadata field, which should be disseminated across the cluster with eventual consistency.

Getting Started

First you should create a new UdpConfig object:

from swimprotocol.udp import UdpConfig

config = UdpConfig(local_name='127.0.0.1:2001',
                   local_metadata={'name': b'one'},
                   peers=['127.0.0.1:2002'])

All other config arguments have default values, which are tuned somewhat arbitrarily with a small cluster of 3-4 members in mind.

Now you can create the cluster members manager and transport layer, and enter the event loop:

from contextlib import AsyncExitStack
from swimprotocol.members import Members
from swimprotocol.udp import UdpTransport

transport = UdpTransport(config)
members = Members(config)
async with AsyncExitStack() as stack:
    worker = await stack.enter_async_context(transport.enter(members))
    await worker.run()  # or schedule as a task

These snippets demonstrate the UDP transport layer directly. For a more generic approach that uses argparse and load_transport, check out the demo.

Checking Members

The Members object provides a few ways to check on the cluster and its members:

for member in members.non_local:
    # all other known cluster members
    print(member.name, member.status, member.metadata)

from swimprotocol.status import Status
for member in members.get_status(Status.AVAILABLE):
    # all cluster members except offline
    print(member.name, member.status, member.metadata)

Alternatively, listen for status or metadata changes on all members:

from swimprotocol.member import Member

async def _updated(member: Member) -> None:
    print('updated:', member.name, member.status, member.metadata)

async with AsyncExitStack() as stack:
    # ...
    stack.enter_context(members.listener.on_notify(_updated))

UDP Transport Security

The UdpTransport transport layer (the only included transport implementation) uses salted hmac digests to sign each UDP packet payload. Any UDP packets received that are malformed or have an invalid signature are silently ignored. The eventual consistency model should recover from packet loss.

The signatures rely on a shared secret between all cluster members, given as the secret=b'...' argument to the Config constructor. If secret=None is used, it defaults to uuid.getnode() but this is not secure for production setups unless all sockets are bound to a local loopback interface.

The cluster member metadata is not encrypted during transmission, so only private networks should be used if metadata includes any secret data, or that secret data should be encrypted separately by the application. Also be aware that low MTU sizes on public networks may affect the ability to synchronize larger amounts of metadata.

Development

First off, I suggest activating a venv. Then, install the development requirements and a local link to the swim-protocol package:

$ pip install -r requirements-dev.txt

Type Hinting

This project makes heavy use of Python's type hinting system, with the intention of a clean run of mypy:

$ mypy

No code contribution will be accepted unless it makes every effort to use type hinting to the extent possible and common in the rest of the codebase.

MIT License

Copyright (c) 2021 Ian Good

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.