universeos 0.1.1

Real-Time LLM Traffic Shadowing Platform SDK

UniverseOS

High-Performance Traffic Shadowing & Experimentation Platform

UniverseOS is a specialized infrastructure layer designed for safe, real-time experimentation with production traffic. While currently optimized for Large Language Models (LLMs), its core architecture provides a generic, high-concurrency shadowing gateway that enables "Parallel Universe" testing—allowing engineers to validate new models, prompts, or configurations against live production data with zero impact on end-user latency or reliability.

Core Architecture

UniverseOS is built on a split-plane architecture designed for scale and resilience:

• Data Plane (C++ Gateway): A high-performance, non-blocking reverse proxy written in C++. It utilizes asynchronous I/O to handle high throughput with minimal overhead. The Gateway intercepts incoming requests and mirrors them to shadow backends asynchronously, ensuring the primary response path remains unaffected.
• Control Plane (Registry & Policy): A dynamic service discovery and routing engine. It allows for hot-swapping of shadow models and granular traffic routing policies (e.g., "shadow 10% of traffic to Model B") without restarting the gateway.
• Observability Plane: A dedicated metrics ingestion pipeline that captures latency, throughput, and model-specific telemetry (e.g., token usage) for side-by-side performance comparison.

Key Features

• Zero-Latency Shadowing: The shadowing mechanism is completely decoupled from the primary request path. Shadow responses are processed out-of-band, guaranteeing no latency penalty for the end user.
• Language Agnostic Integration: Designed as a network-level infrastructure component. While a Python SDK is provided for convenience, the system works with any client capable of making HTTP requests.
• Production Safety: Failures in shadow models are isolated. If a shadow model crashes or times out, the primary request completes successfully, and the error is logged for analysis.
• Live A/B Testing & Evaluation: Facilitates "dark launching" of new models. Compare the quality and cost of a new model against the production baseline using real-world inputs before exposing it to users.

Quickstart

1. Deploy the Control Plane

The core infrastructure runs as a set of containerized services.

docker-compose up -d

2. Install the Python SDK

For Python applications, the SDK provides a seamless integration point.

pip install universeos

3. Integration

Wrap your existing API calls to enable automatic shadowing. The SDK handles the communication with the Gateway.

from universeos import universe_shadow, init_universe

# Initialize connection to the sidecar/gateway
init_universe()

@universe_shadow
def generate_response(prompt):
    # Your existing production logic (e.g., OpenAI, Anthropic, or local LLM)
    return production_client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}]
    )

Technical Specifications

• Gateway: C++17, Asynchronous Socket I/O (poll/epoll)
• Protocol: HTTP/1.1 (Streaming Support)
• Configuration: Dynamic YAML-based policy loading
• SDK: Python 3.7+ (Thread-safe, minimal dependencies)

Building from Source

To build the high-performance Gateway and Control Plane services from source:

mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j4