language-pipes 0.19.6

A privacy focused distributed algorithm for llm inference

Language Pipes (Beta)

A privacy focused distributed algorithm for llm inference

Language Pipes is an open-source distributed network application designed to increase access to local language models by allowing for privacy protected computation between peer to peer nodes.

Disclaimer: This software is currently in Beta. Please be patient and if you encounter an error, please fill out a github issue!

---

Features

• Quick Setup
• OpenAI compatible API
• Privacy-focused architecture
• Decentralized peer to peer network
• Download and use models by HuggingFace ID

---

What Does Language Pipes do?

Large Language models work by passing information through many layers. At each layer, several matrix multiplicatitons between the layer weights and the system state are performed and the data is moved to the next layer. Language pipes works by hosting different layers on different machines to split up the RAM cost across the system. This project contrasts with existing programs like vLLM by focusing on decentralization and privacy.

Here are some helpful links to get started:

• To learn more about the privacy protecting mechanism click here.
• To learn more about how language pipes works click here.
• To learn about how Language Pipes processes jobs click here.

Installation

Ensure that you have Python installed.

If you need gpu support, first make sure you have the correct pytorch version installed for your GPU's Cuda compatibility using this link:
https://pytorch.org/get-started/locally/

To download the models from Huggingface, ensure that you have git and git lfs installed.

To start using the application, install the latest version of the package from PyPi.

Using Pip:

pip install language-pipes

Quick Start

The easiest way to get started is with the interactive setup wizard:

language-pipes

This launches a menu where you can create, view, and load configurations. Select Create Config to walk through the setup wizard, which guides you through your first configuration. After creating a config, select Load Config to start the server.

We also support loading toml files directly!
If you need help loading them read the CLI documentation here.

---

Two Node Example

This example shows how to distribute a model across two computers using the interactive wizard.

Node 1 (First Computer)

Start language pipes:

language-pipes

Prompt	Value	Description
Node ID	node-1	Unique identifier for this node on the network
Model ID	Qwen/Qwen3-1.7B	HuggingFace model to load
Device	cpu	Hardware to run inference on
Max memory	1	GB of RAM to use (loads part of the model)
Load embedding/output layers	Y	Required for the first node to handle input/output
Enable OpenAI API	Y	Exposes the OpenAI-compatible endpoint
API port	8000	Port for the API server
First node in network	Y	This node starts the network
Encrypt network traffic	N	Disable encryption for simplicity

Node 2 (Second Computer)

Start language pipes with this command:

language-pipes

Prompt	Value	Description
Node ID	node-2	Unique identifier for this node on the network
Model ID	Qwen/Qwen3-1.7B	Must match the model on node-1
Device	cpu	Hardware to run inference on
Max memory	3	GB of RAM to use (loads remaining layers)
Load embedding/output layers	N	Node-1 already handles these
Enable OpenAI API	N	Only node-1 needs the API
First node in network	N	This node joins an existing network
Bootstrap node IP	192.168.0.10	Node-1's local IP address
Bootstrap port	5000	Node-1's network port
Encrypt network traffic	N	Must match node-1's setting

Node-2 connects to node-1 and loads the remaining model layers. The model is now ready for inference!

Test the API

The model is accessible via an OpenAI-compatible API. Using the OpenAI Python library:

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",  # node-1 IP address
    api_key="not-needed"  # API key not required for Language Pipes
)

response = client.chat.completions.create(
    model="Qwen/Qwen3-1.7B",
    max_completion_tokens=100,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a haiku about distributed systems."}
    ]
)

print(response.choices[0].message.content)

Install the OpenAI library with: pip install openai

To learn about how to work with the Open AI compatable server click here.

Model choice

Currently Language Pipes targets the Qwen3 and Qwen3-moe architectures.

Future Updates

There are plans to update the project in the future if it gets enough traction. These improvements include:

• More models supported
• 8 bit and 4 bit quantization support (currently everything is run in fp16)
• GGUF support (currently everything needs to be in safetensors format)
• Responses endpoint (currently only /v1/chat/completions is supported)
• huggingface library support for downloading models that require authentication (currently git-lfs)

So please star the repo if you find it useful :)

Dependencies

• pytorch
• transformers

Documentation

• CLI Reference
• Privacy Protection
• Configuration Manual
• Architecture Overview
• Open AI Compatable API
• Job Processor State Machine
• The default peer to peer implementation
• The way Language Pipes abstracts from model architecture