connex-ai 0.2.1

Endpoint mobility primitives for dynamic LLM serving
communication endpoints that join, leave, fail and migrate with traffic in flight

Connex is a communication library for distributed LLM serving in which the set of peers keeps changing: replicas scale up and down, processes die and come back, work moves between instances. Every endpoint has a stable logical identity that a control plane resolves to its current address, and senders keep in-flight work correct when the endpoint behind an identity changes. Requests hand over to another replica on failure, and new replicas take traffic as soon as they join. Connex is the system described in our SIGCOMM 2026 paper (Citation).

Supported scenarios

PP, PD, TP, EP, RL and agent support run on upstream engines as plugins, without a fork. Upstream PRs: vLLM ConnexConnector for PD (vllm#63804), vLLM Connex weight transfer backend for RL (vllm#63805), SGLang connex KV transfer backend (sglang#46599), vLLM pluggable PP group via --pp-group-cls (vllm#63897), vLLM --pipeline-stage-rank for external PP stages (vllm#63898), vLLM recompute of lost KV requests (vllm#63899). Pipeline parallelism is proposed in vllm#63818 (RFC: pluggable PP transport); the plugin connex.adapters.vllm.pp runs on the latest vLLM main with the three changes it proposes, and the Docker image builds that by default (integrations/vllm/README.md, results in docs/results.md). Limits per scenario: Scenarios and status.

Quick start

pip install connex-ai                 # core (import name: connex)
pip install "connex-ai[rs]"           # + connex-rs, the RDMA data plane
pip install "connex-ai[vllm,rs]"      # + vLLM plugins (PP, PD, TP, EP, RL)
pip install "connex-ai[sglang,rs]"    # + SGLang plugins

A CPU-only example: a sender talks to two replicas of one logical group and keeps working while one dies, rejoins, migrates to a new address and leaves (examples/mobile_endpoints.py):

peers = ElasticPeerSet(owner, transport, stage_idx=1)   # resolved through the directory
peers.send(peers.pick(), b"step")
peers.mark_failed(("demo", 1, 0, 1))    # replica 1 died
peers.send(peers.pick(), b"step-42")    # in-flight step handed over to replica 0
b.register()                            # replica 1 rejoins
a.register(addresses=[new_addr_a])      # replica 0 migrates, same identity

Prefill-decode disaggregation on upstream vLLM:

# prefill and decode instances; the connector is registered by the connex-ai plugin
vllm serve <model> --port 8100 --kv-transfer-config '{"kv_connector":"ConnexConnector",
  "kv_role":"kv_producer","kv_connector_extra_config":{"ctrl_port":17100}}'
vllm serve <model> --port 8200 --kv-transfer-config '{"kv_connector":"ConnexConnector",
  "kv_role":"kv_consumer"}'
connex-pd-router --port 8000 --prefill http://<p-host>:8100 --decode http://<d-host>:8200

Every scenario has its own guide (links in the table above). Deployment from a CPU demo to Kubernetes: docs/deploy.md.

How it works

Connex separates who talks to whom, and what happens when that changes from how the bytes move. The control plane holds logical identities, leases and membership events. The delivery contract (correlation ids, idempotent retries, stale-reply filtering) makes it safe to resend in-flight work to another replica. Bytes move through a pluggable data plane: on RDMA hosts the default is connex-rs, our Rust data plane over UCX, which reports a dead peer within milliseconds and lets a restarted process rejoin under the same identity. Mooncake Transfer Engine, UCX and ZeroMQ also work. Details: docs/overview.md.

Results

Llama-3-8B on A40 GPUs (OLMoE-1B-7B for EP); every request completed and no token differed from the greedy reference in any failure run.

Full tables, setup and reproduction commands: docs/results.md (pipeline on upstream vLLM: section 13).

Documentation

Contributing

See CONTRIBUTING.md. Changes are listed in CHANGELOG.md.

Citation

If you use Connex in your research, please cite our paper:

@inproceedings{lin2026connex,
  title={Connex: Endpoint Mobility Primitives for Dynamic LLM Serving},
  author={Lin, Yanying and Liu, Vincent and Luo, Tao and Xu, Chengzhong and Ye, Kejiang},
  booktitle={Proceedings of the ACM SIGCOMM 2026 Conference},
  pages={46--59},
  year={2026}
}

License

Apache-2.0, see LICENSE.