Multiplexer

A lightweight TCP message broker for service-to-service messaging. Not a socket multiplexer, not a WebSocket server: it is the broker your services talk through, on a trusted network, for request/reply against a pool of backends and publish/subscribe of events.

Peers connect over TCP, announce their peer type, and exchange typed messages with low latency. A declarative rules file defines the routing for each message type: round-robin load balancing across a pool of backends, fan-out to every subscriber, or direct addressing by instance id. It is stateless by design: no queues, no persistence, no authentication, and no coordination between instances. It terminates no HTTP or WebSocket connections and manages no client sessions; a web server is one kind of peer among others.

pip install mx-multiplexer installs the Python package and the multiplexer itself, mxcontrol; a release also ships mxcontrol as one binary for any Linux with glibc 2.28 or newer and as a container image, and the C++ library as a Debian or Ubuntu package, all for Linux x86_64, and the Rust library is the crate mx-multiplexer. Nothing needs building; the C++17 source builds with Bazel or make when you want to. In Docker a multiplexer is one docker run, and on Kubernetes a StatefulSet behind a headless Service, with no operator, no leader and no state to keep (in Docker and on Kubernetes). Deploy several instances active-active: there is no single point of failure, every client and backend connects to all of them, and failover between them is automatic and transparent.

Multiplexer has been used in production in a number of projects since 2008. This repository adds what publication calls for: complete documentation, a test for every failure mode, and static thread-safety analysis.

Quick start

On Linux x86_64 with Python 3.10 or newer: the package from PyPI, which brings the multiplexer with it, a rules file, two multiplexers, a backend and a client. Nothing to build.

python3 -m venv mx && . mx/bin/activate
pip install mx-multiplexer
mxcontrol generate_rules multiplexer.rules          # the system rules every rules file starts from
cat >> multiplexer.rules <<'END'
peer { type: 200 name: "WEB" }
peer { type: 201 name: "COMPUTE" }
type { type: 301 name: "COMPUTE_REQUEST" to { peer: "COMPUTE" whom: ANY } }
type { type: 302 name: "COMPUTE_RESPONSE" }
END
mxcontrol generate_constants multiplexer.rules --python multiplexer_constants.py
mxcontrol run_multiplexer --address 127.0.0.1:1980 2> mx1.log &
mxcontrol run_multiplexer --address 127.0.0.1:1981 2> mx2.log &

backend.py answers each request with its payload upper-cased:

from multiplexer.servers import BaseMultiplexerServer
from multiplexer_constants import peers, types


class Compute(BaseMultiplexerServer):
    def handle_message(self, mxmsg):
        self.send_message(message=mxmsg.message.upper(), type=types.COMPUTE_RESPONSE)


Compute([("127.0.0.1", 1980), ("127.0.0.1", 1981)], type=peers.COMPUTE).serve_forever()

client.py asks:

from multiplexer.threaded_client import ThreadedClient
from multiplexer_constants import peers, types

client = ThreadedClient([("127.0.0.1", 1980), ("127.0.0.1", 1981)], type=peers.WEB)
print(client.query(b"hello", type=types.COMPUTE_REQUEST, timeout=10).message)
client.shutdown()

Start the backend, then ask, take a multiplexer away and ask again:

$ python backend.py 2> backend.log &
$ python client.py 2> client.log
b'HELLO'
$ kill %1                       # the multiplexer on 1980
$ python client.py 2> client.log
b'HELLO'

Both peers are connected to both multiplexers, so the second query goes through the one left, and nothing waited or failed. kill %2 %3 ends the rest. The libraries log their connections to stderr, which the commands above send to files.

In Docker and on Kubernetes

A multiplexer is one process, one rules file and nothing to keep, so a container is all it takes. The image, ghcr.io/kulewski/multiplexer, is 32 MB, an 8 MB download: mxcontrol and the system rules on distroless's glibc base, running as nonroot, listening on 1980 and reading /etc/mx/multiplexer.rules. Mount the quick start's directory there, and these two lines replace the quick start's two mxcontrol run_multiplexer lines:

docker run -d --name mx1 -p 1980:1980 -u "$(id -u)" -v "$PWD:/etc/mx:ro" ghcr.io/kulewski/multiplexer:latest
docker run -d --name mx2 -p 1981:1980 -u "$(id -u)" -v "$PWD:/etc/mx:ro" ghcr.io/kulewski/multiplexer:latest

The backend and the client stay as they are, and docker stop mx1 is the quick start's kill %1; -u runs the containers as you, so that they can read your directory. An edit of multiplexer.rules is put in use by both within seconds, with no restart; the directory is mounted rather than the file so that an editor's save reaches them (changing the rules).

On Kubernetes there is nothing to operate: no operator, no leader, no shared state. A StatefulSet behind a headless Service gives each multiplexer a name of its own, mx-0.mx, mx-1.mx, mx-2.mx, every peer is given all of them, and the rules file is a ConfigMap:

kubectl create configmap mx-rules --from-file=multiplexer.rules
kubectl apply -f mx.yaml

# mx.yaml
apiVersion: v1
kind: Service
metadata: {name: mx}
spec:
  clusterIP: None              # headless: a DNS name per pod, no load balancing
  selector: {app: mx}
  ports: [{port: 1980}]
---
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: mx}
spec:
  serviceName: mx
  replicas: 3
  selector: {matchLabels: {app: mx}}
  template:
    metadata: {labels: {app: mx}}
    spec:
      containers:
        - name: mx
          image: ghcr.io/kulewski/multiplexer:latest   # or a version, to pin it
          ports: [{containerPort: 1980}]
          readinessProbe: {tcpSocket: {port: 1980}, periodSeconds: 5}
          volumeMounts: [{name: rules, mountPath: /etc/mx}]
      volumes: [{name: rules, configMap: {name: mx-rules}}]

The peers get mx-0.mx:1980,mx-1.mx:1980,mx-2.mx:1980. A rolling update restarts the multiplexers one at a time, which costs a request nothing while the peers are on the others, and an edit of the ConfigMap needs no rollout at all. docs/operations.md has the complete manifest, with the pods spread over nodes and a PodDisruptionBudget, and the inference example's compose.yaml runs a whole stack with docker compose up: two multiplexers, the workers and the web app (examples/inference).

Internal service-to-service messaging

An example: web servers on the left, two multiplexers in the middle, a pool of C++ backends doing the expensive computation on the right. The web servers are peers like any other, here the ones that ask; their own HTTP or WebSocket traffic with browsers is theirs and never reaches the multiplexer. Every peer is connected to both multiplexers; a request from any web server reaches one backend, chosen round robin, and the answer comes back to the server that asked. Lose either multiplexer, or any backend, and nothing stops.

The rules behind the picture, in the format rules.md describes, are the quick start's: the system rules every rules file starts from, left out here, then the four entries it appended, written out:

# ... the system rules, as `mxcontrol generate_rules` writes them ...

peer {
    type: 200
    name: "WEB"
}

peer {
    type: 201
    name: "COMPUTE"
}

type {
    type: 301
    name: "COMPUTE_REQUEST"
    to {
        peer: "COMPUTE"
        whom: ANY
    }
}

type {
    type: 302
    name: "COMPUTE_RESPONSE"
}

Every peer has a peer type and every message a message type, numbers the rules file names. A type's rule says where its messages go: to ANY one peer of a type, round robin, or to ALL of them, and several rules fan one message out to several types. A message with to set goes to that one instance whatever the rules say, which is how a reply finds the peer that asked, so COMPUTE_RESPONSE needs no rule. A deployment keeps its own rules file, the system rules and its own types after them, and the multiplexers put a changed file in use without a restart (changing the rules); the constants a program imports are generated from it, by mxcontrol generate_constants or a Bazel build. rules.md describes every field and docs/README.md the words.

Highlights

• Highly available, no single point of failure. Run several multiplexers active-active; clients and backends connect to all of them, fail over automatically, and reconnect on their own. A rolling restart of three multiplexers under traffic costs no request more than a millisecond (rolling_restart).
• One docker run, or a StatefulSet on Kubernetes. The image is 32 MB, mxcontrol and the system rules on distroless, running as nonroot; mount a rules file over its own and it is a multiplexer. On Kubernetes a StatefulSet behind a headless Service runs as many as you want, with no operator, no leader and no shared state, and an edit of the rules' ConfigMap is put in use without a rollout (in Docker and on Kubernetes).
• Fault-tolerant requests. A request whose backend dies, or whose connection dies, is retried once, through the backend a search across every multiplexer finds, all inside one call (how a query is answered).
• Zero-downtime deployments. A backend asked to leave drains: it tells every multiplexer to route it nothing new, finishes what it holds, and leaves as soon as they have confirmed, so a Kubernetes rolling restart with a preStop hook costs nobody a timeout, or a retry (backend_drains).
• Load balancing and fan-out from one rules file. Round-robin across a backend pool, publish/subscribe to every subscriber, or direct addressing, chosen per message type, no code change (rules); the file is edited under running multiplexers, which put it in use without a restart (changing the rules).
• Pools within a type, by key. A message may say what it is about, a tenant, a model, a group, and each backend which keys of a type it takes: a tenant's requests reach only the backends that take its key, an event only the subscribers of its group, and a failover's retry stays with the backends that take the key (keys, examples/models).
• One instance, one path, when it matters. A request addressed to one backend instance reaches it or fails, never another instance, and is found again behind another multiplexer; a lane keeps a stream of messages on one connection, in order, following a failover or pinned (lanes, pinning and addressed queries).
• High performance, low latency. One thread and one event loop per multiplexer, frames forwarded as they arrived, never re-serialized, one write per frame with Nagle's algorithm off on every connection: over 120,000 request/reply round trips per second through one multiplexer on 70% of one core, 17 µs per round trip when idle, 7 MB resident, measured with tests/bench.sh on a Threadripper PRO 9965WX, and as fast with Rust peers as with C++ ones.
• Stateless and lightweight. No external coordination service, no persistence, one binary and one rules file; scale horizontally by adding instances and backends. A documented binary protocol over TCP with Protocol Buffers (wire format).
• Connections for C++, Python and Rust, and asyncio. SyncClient for a program on one thread; ThreadedClient, with callback-based asynchronous queries, for servers, safe to share between threads; and AsyncClient for asyncio programs, which awaits queries and sends and delivers events to coroutines on the loop; all fork-aware and clean at interpreter exit (Python API, C++ API). In Rust the same classes, ThreadedClient's calls awaited on any executor, tokio's included (Rust API).
• Two server classes that run the loop for you. BaseMultiplexerServer runs the loop and your handle_message() on one thread, for quick handlers; BaseThreadedMultiplexerServer runs handlers on worker threads behind a heartbeating io thread, for requests that take minutes, several at once, or a handler that blocks on a query of its own. Either is the usual base of a backend; a backend can also be built on ThreadedClient or AsyncClient (which to use).
• Tested for every failure mode. One documented integration scenario per failure, AddressSanitizer, ThreadSanitizer, LeakSanitizer, clang thread-safety analysis and a soak test (scenarios).
• Test infrastructure you can import. The harness those scenarios run on is a public package: real multiplexers on ephemeral ports in one line, scripted backends and clients in-process for unit tests, peers as processes for end-to-end tests, and a Bazel macro that runs a scenario against the shipped roles or a binary of your own (testing).
• See what was routed, when asked. A peers file lists who is connected. A recording holds every routed message, with who sent it, who received it or why nobody did; switched on at start, or started, stopped and tapped live on a running cluster from one command, with the operator's consent given as a flag. A session replays offline and scripts test peers from real traffic (operations).

How it compares

• RabbitMQ, Kafka: those persist and queue; the multiplexer does neither. A message goes to whoever is connected right now or is dropped, at most once per multiplexer, which is what makes it stateless and simple to run.
• ZeroMQ: a library without a broker, so peers need each other's addresses; here every peer knows only the multiplexers, and routing is configuration.
• gRPC: point-to-point calls with generated stubs per service; here one rules file routes typed messages to pools, subscribers or instances, and a request finds a live backend on its own.
• NATS: the closest in spirit. NATS servers form a cluster; multiplexers do not know each other, and the clients provide the redundancy by connecting to all of them.

Semantics: delivery, failures and defaults states exactly what is promised, for the deployment the multiplexer is designed for: several of them, every peer connected to all.

Examples

Complete programs under examples/, each with a README that runs it and a walkthrough that builds it from nothing, echo's apart, which runs it step by step. Four small ones first, then building blocks, then what the broker's latency makes possible:

• echo: a backend and a client, in Python and in C++, and threads sharing one client.
• echo-rust: the same backend and client in Rust, the backend draining on SIGTERM.
• aio: an asyncio TCP gateway in front of a chat backend, a reply to the one who asked and a broadcast to everyone.
• health: a backend's health check that sees its serve loop, a Kubernetes liveness probe beside the drain on a preStop hook.
• cache: a replicated cache as a Django cache backend, a write to every replica and a read from any, and a journal added by changing the rules under the running multiplexers.
• channels: a Django Channels channel layer, Redis replaced by one setting.
• inference: a web app asking a pool of PyTorch workers, with a worker killed, a rolling restart and a multiplexer killed, in an executed notebook.
• models: workers that each hold some of the models, a request's key naming its model and each worker's key filter the models it holds, and a model loaded, and unloaded while its requests keep coming.
• stream: an answer that arrives token by token, as a language model's does, one request and one reply with the pieces addressed and numbered between them, a multiplexer killed under it and the lost pieces asked for again, and the answer as Server-Sent Events.
• audio: an audio room, a query per 10 ms frame to a C++ worker and the answer heard by everyone, the worker replaced under the stream.

Using it from Python

pip install mx-multiplexer installs the package, its compiled extension included, and the mxcontrol command, which runs the multiplexer and its tools, on Linux x86_64 for CPython 3.10 and newer; the import is multiplexer, since multiplexer on PyPI is an unrelated package. mxcontrol generate_rules your.rules writes the system rules every rules file starts from; add your own types after them, and mxcontrol generate_constants your.rules --python multiplexer_constants.py --pyi multiplexer_constants.pyi writes their constants, once and again whenever the file changes, for from multiplexer_constants import peers, types. A Bazel build generates them on its own.

The quick start's two classes are the usual start: a backend built on BaseMultiplexerServer, whose serve_forever() runs the loop and calls handle_message for each request, send_message replying to the peer that asked, and a ThreadedClient, safe to share between threads, whose query() sends a request and returns the answer. An event is sent the same way without waiting for an answer, client.send_message(b"payload", type=types.SOME_EVENT), and to=<an instance id> addresses one peer, bypassing the routing rules. SyncClient (named Client before 2.4.0), AsyncClient for asyncio and BaseThreadedMultiplexerServer, whose handlers run on worker threads, are in the Python API, and which class to build on compares them all. A test starts real multiplexers with multiplexer.testing, which runs the package's mxcontrol (testing).

Using it from C++

pip installs no C++ library: the Debian and Ubuntu packages on the release page hold it, its headers and a pkg-config file, besides mxcontrol, one package per release, because protobuf's C++ library promises no compatibility between versions and the library is built against each release's own. With the package installed, and libprotobuf-dev and libasio-dev:

sudo apt install ./multiplexer_<version>~<codename>_amd64.deb
g++ -std=c++17 backend.cc $(pkg-config --cflags --libs multiplexer) -o backend

The C++ library has the Python package's classes but the asyncio one: SyncClient with query, schedule_one and schedule_all, ThreadedClient, BaseMultiplexerServer and BaseThreadedMultiplexerServer; examples/echo/backend.cc is a complete backend and the C++ API describes them. The constants of your own rules file come from mxcontrol generate_constants your.rules --cxx multiplexer/multiplexer.constants.h, placed on the include path before the package's copy (packaging).

Using it from Rust

The crate mx-multiplexer is the Rust library, written in Rust on mio rather than bound to the C++ one, with the classes of the other two and their behaviour: SyncClient, ThreadedClient, whose calls are also awaited on any executor, tokio's included, with no runtime of its own, and the server classes BaseMultiplexerServer and BaseThreadedMultiplexerServer, which serve a Handler. cargo add mx-multiplexer adds it, Rust 1.85 or newer on Linux; the constants of your own rules file come from mxcontrol generate_constants your.rules --rust src/multiplexer_constants.rs, written once and again whenever the file changes. The Rust API describes the classes, and examples/echo-rust is a complete backend and client.

Getting it

Two ways, and the choice is only whether you build it:

• A release. pip install mx-multiplexer fetches the Python package from PyPI, mxcontrol inside, one manylinux wheel per CPython from 3.10. Every release on GitHub ships the same set, built from one commit and stripped: those wheels, mxcontrol for any x86_64 Linux with glibc 2.28 or newer, the container image ghcr.io/kulewski/multiplexer:<version>, also tagged latest, with nothing in it but mxcontrol and the system rules, a Debian package per Debian and Ubuntu release with mxcontrol, the C++ library, its headers and a pkg-config file, and the Rust crate, which crates.io serves as mx-multiplexer. docs/packaging.md says how to use each; in Docker and on Kubernetes runs the image in both, and docs/operations.md has the complete Kubernetes manifest.
• From source, with Bazel, which is how the repository is developed and tested and how another Bazel workspace consumes it, or with make from the distribution's own packages: building from source.

Building from source

Building with Bazel

Prerequisites, verified on clean Debian 12, Ubuntu 24.04 and Debian 13 installations: Bazel 6.2 through Bazelisk, a C++17 compiler, protobuf-compiler with libprotobuf-dev, Python 3.10 or newer with python3-dev and python3-protobuf. Asio and pybind11 are fetched by Bazel. docs/building.md has the exact packages, how to bring your own protobuf, and docker/check.sh, which proves the list on a clean container.

bazel build //...
bazel test //...                              # unit and integration tests
bazel test --test_tag_filters=-slow //...     # the fast ones, a few seconds
bazel run //mxcontrol run_multiplexer
cd examples/echo && bazel test //...          # a backend and a client, built the way your code will be

The integration tests under tests/ start real multiplexers and play peers in Python, C++ and Rust; see tests/README.md. mxcontrol run_multiplexer accepts --address host:0 --port-file PATH to pick a free port and report it, and on SIGTERM sends what it holds, closes its connections and exits with 0.

--config=debug keeps debug info and frame pointers; --config=release builds optimized, stripped binaries. Run the server inside screen or tmux if you want to get back to the session later.

Your own peer and message types

multiplexer.rules in this repository is the system rules, which every rules file starts from. Keep your deployment's rules file in your own repository and point the build at it:

bazel build --//:multiplexer_rules=//your/pkg:multiplexer.rules //...

When this repository is consumed as an external Bazel repository named mx, the flag is --@mx//:multiplexer_rules=.... At run time mxcontrol run_multiplexer reads multiplexer.rules from the current directory unless --rules points elsewhere.

Building without Bazel

For a machine that will not have Bazel, a Makefile builds the same things from the distribution's own compiler, protobuf, Asio and pybind11:

make -j                      # build/bin/mxcontrol, build/libmultiplexer.a with headers, build/python/
make check                   # the C++ and Python unit tests, against what was built
make wheel                   # a pip wheel of the Python package, mxcontrol inside
sudo make install            # mxcontrol, the library and the headers under /usr/local
make RULES=your.rules -j     # the constants from your rules file

docs/building.md lists the packages and what a program links and imports. The test roles, the scenarios, the examples and the sanitizer builds stay with Bazel.

Using it from another Bazel workspace

Declare this repository as mx, then two calls in your WORKSPACE:

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

git_repository(
    name = "mx",
    commit = "<commit sha>",
    remote = "https://github.com/kulewski/multiplexer.git",
)

load("@mx//bazel:deps.bzl", "mx_dependencies")
mx_dependencies()
load("@mx//bazel:setup.bzl", "mx_setup")
mx_setup()

Then depend on @mx//multiplexer:clients (SyncClient) or @mx//multiplexer:servers (BaseMultiplexerServer) from Python, @mx//multiplexer:client from C++; import and include paths are unchanged. A workspace that uses C++ alone calls mx_dependencies(python = False) and no mx_setup(), and fetches no Python rules. examples/ holds complete workspaces built that way, starting with examples/echo. Your tests get @mx//multiplexer/testing: real multiplexers on ephemeral ports, scripted peers, and the macro this repository's own scenarios use (docs/api_python.md).

Where to read next

docs/README.md defines the terminology, shows the reference deployment and traces a query and an event step by step. The examples carry on from the quick start: cache builds a pip project from nothing, and echo runs the smallest program with Bazel and reads what each side prints. The reference pages under docs/ cover the rules file, the client libraries, mxcontrol, the wire format and operations.

History

Multiplexer was built at Azouk Network Ltd. in 2008-2009, first as a Python prototype by Krzysztof Kulewski and then as the C++ implementation this repository grew out of, by Piotr Findeisen. Azouk released it under the MIT license, and Krzysztof Kulewski has owned and developed it since 2010, in production use throughout. See AUTHORS.

License

MIT, see LICENSE. What the binaries hold of other projects, compiled in, is in THIRD_PARTY_NOTICES.