universeos 0.1.0

Real-Time LLM Traffic Shadowing Platform SDK

UniverseOS 🌌

Real-Time LLM Traffic Shadowing Platform

UniverseOS is a drop-in library for safe, real-time experimentation with Large Language Models. It allows you to shadow production traffic to experimental models ("parallel universes") without impacting user latency or reliability.

🚀 Quickstart

1. Start the Control Plane

docker-compose up -d

2. Install the SDK

pip install universeos

3. Instrument Your Code

from universeos import universe_shadow, init_universe

init_universe()

@universe_shadow
def chat(prompt):
    # Your existing OpenAI/LLM call
    return openai.ChatCompletion.create(model="gpt-4", ...)

🌟 Features

• Zero-Latency Overhead: Shadowing happens asynchronously.
• Dynamic Routing: Add/remove shadow models instantly via Registry.
• Causal Policy Engine: Intelligent traffic routing based on real-time feedback.
• Observability: Built-in metrics for latency, cost, and quality.

📂 Structure

• sdk/python: Python Client SDK.
• services/gateway: High-performance C++ Proxy.
• services/policy: Bandit Algorithm Engine.
• services/registry: Service Discovery.
• services/metrics: Telemetry Ingestion.

🛠 Building from Source

mkdir build && cd build
cmake ..
make