T-LEX Edge

Edge-Decoupled LLM Inference - Run 32B+ models on ANY device!

The Problem

Running large language models locally requires expensive GPU hardware:

Most edge devices (laptops, IoT, drones, phones) can't run these models.

The Solution: Split-Brain Architecture

T-LEX separates inference (GPU server) from decoding (edge device):

┌─────────────────────────┐         ┌─────────────────────────┐
│     GPU Server          │         │     Edge Device         │
│  ┌─────────────────┐    │ tokens  │  ┌─────────────────┐    │
│  │ Ollama/OomLlama │────┼────────►│  │ T-LEX Decoder   │    │
│  │ 32B model       │    │         │  │ vocab.db (6MB)  │    │
│  │ 24GB VRAM       │    │         │  │ NO GPU!         │    │
│  └─────────────────┘    │         │  └─────────────────┘    │
└─────────────────────────┘         └─────────────────────────┘

Performance

Result: 8x faster with zero GPU on edge!

Installation

# Basic installation
pip install tlex-edge

# With server support (FastAPI)
pip install tlex-edge[server]

# With RAG support (ChromaDB)
pip install tlex-edge[rag]

# Full installation
pip install tlex-edge[full]

Quick Start

Python API

from tlex import TLexClient, EdgeDecoder

# Connect to remote GPU server
client = TLexClient("http://gpu-server:11434")

# Generate with 32B model - no local GPU needed!
result = client.generate(
    "Explain quantum computing",
    model="humotica-32b",
    max_tokens=100
)
print(result.text)
print(f"Speed: {result.tokens_per_second:.1f} tok/s")

# Streaming output
for chunk in client.stream("Tell me a story"):
    print(chunk, end="", flush=True)

Command Line

# Generate text
tlex generate "What is AI?" --model qwen2.5:7b --server http://gpu-server:11434

# Interactive chat
tlex chat --model humotica-32b

# List available models
tlex models

# Benchmark decoder
tlex benchmark --vocab qwen_vocab.db

Docker

# Build
docker build -t tlex-edge .

# Run
docker run -it tlex-edge generate "Hello!" --model qwen2.5:7b

# With docker-compose
docker-compose up -d
docker-compose exec tlex chat

Building Vocabulary Database

The vocab database is all an edge device needs to decode tokens:

# From command line
tlex vocab Qwen/Qwen2.5-7B-Instruct --output qwen_vocab.db

# From Python
from tlex import build_vocab_db
build_vocab_db("Qwen/Qwen2.5-7B-Instruct", "qwen_vocab.db")

Size comparison:

• Qwen 7B model: ~14 GB
• qwen_vocab.db: ~6 MB (2300x smaller!)

Server Setup

T-LEX works with any Ollama-compatible server:

# On your GPU server (P520, etc.)
ollama serve

# Pull models
ollama pull qwen2.5:7b
ollama pull qwen2.5:32b

Architecture

              ┌─────────────────────────────────────────────────┐
              │                  GPU SERVER                      │
              │  ┌─────────────────────────────────────────┐    │
              │  │  Ollama / OomLlama                      │    │
              │  │  - Qwen 7B/32B/72B                      │    │
              │  │  - LLaMA 70B                            │    │
              │  │  - Any GGUF model                       │    │
              │  └─────────────────────────────────────────┘    │
              │                      │                           │
              │                      │ HTTP Stream               │
              │                      │ (token chunks)            │
              └──────────────────────┼───────────────────────────┘
                                     │
                    ┌────────────────┴────────────────┐
                    │           NETWORK               │
                    │    (LAN / Internet / I-Poll)    │
                    └────────────────┬────────────────┘
                                     │
┌────────────────────────────────────┴────────────────────────────────────┐
│                           EDGE DEVICES                                   │
│                                                                          │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐ │
│  │   Laptop     │  │  Raspberry   │  │    Phone     │  │    Drone     │ │
│  │              │  │     Pi       │  │              │  │              │ │
│  │ vocab.db 6MB │  │ vocab.db 6MB │  │ vocab.db 6MB │  │ vocab.db 6MB │ │
│  │   NO GPU!    │  │   NO GPU!    │  │   NO GPU!    │  │   NO GPU!    │ │
│  └──────────────┘  └──────────────┘  └──────────────┘  └──────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘

Use Cases

• IoT Devices: Smart home with AI, no cloud dependency
• Drones: On-board AI decisions, low latency
• Mobile Apps: Full LLM power without draining battery
• Air-gapped Networks: Self-hosted inference + edge decode
• Cost Reduction: One GPU server, unlimited edge clients

Part of HumoticaOS

T-LEX is part of the HumoticaOS ecosystem:

• TIBET: Trust & provenance for AI actions
• I-Poll: AI-to-AI messaging
• OomLlama: Native Rust inference engine

One love, one fAmIly! 🦙❤️

License

MIT License - see LICENSE for details.

Contributing

Contributions welcome! Please read CONTRIBUTING.md first.