vortexrt 0.3.1

Unified AI model inference — auto-detect, auto-route, auto-serve with multi-engine priority routing.

VortexRT

Unified AI Model Inference — One Library, Every Model

---

VortexRT is a Python library for AI model inference. Give it a model name, it detects the task, picks the best engine for your hardware, installs missing dependencies, and runs inference.

from vortexrt import VortexRT
vrt = VortexRT()
llm = vrt.load("Qwen/Qwen2.5-1.5B-Instruct")
result = vrt.predict(llm.id, {"prompt": "Explain quantum computing"})

---

Engines

VortexRT is built on 7 open-source inference engines. Each engine is independently licensed and can be used in any other project.

Engine	License	Specialisation	% of Tasks
Transformers	Apache 2.0	Text, classification, QA, NER, embeddings, TTS, ASR	45%
Diffusers	Apache 2.0	Text-to-image, image-to-image	10%
ONNX Runtime	MIT	Custom ONNX, audio, TTS, ASR, DirectML	15%
vLLM	Apache 2.0	High-throughput LLM (PagedAttention)	10%
vLLM-Omni	Apache 2.0	Accelerated image/video (DiT, Cache-DiT)	10%
llama.cpp	MIT	GGUF LLMs on CPU	5%
Stable Diffusion.cpp	MIT	GGUF diffusion on CPU/GPU	5%

All engines are open source and permissible for commercial use. Our library is MIT licensed — compatible with all of them.

Hardware Support

	NVIDIA CUDA	AMD ROCm	Intel XPU/Arc	CPU Only	Apple MPS
Transformers	✓	✓	✓	✓	✓
Diffusers	✓	—	—	✓	✓
ONNX Runtime	✓ CUDA EP	✓ ROCm EP	✓ OpenVINO EP	✓	✓ CoreML
vLLM	✓	✓	✓	✓ (slow)	—
vLLM-Omni	✓	✓	✓	—	—
llama.cpp	✓ Vulkan	✓ Vulkan	✓ Vulkan	✓	✓ Metal
diffusion-gguf	✓ Vulkan	✓ Vulkan	✓ Vulkan	✓	—

Yes, it works on AMD, Intel, and CPU. The library automatically detects your hardware and picks the best available engine. If vLLM isn't available (no NVIDIA GPU), it falls back to Transformers for text and Diffusers for images.

Video Generation

Video generation is registered as a task (text-to-video) but requires a video-capable model and engine. vLLM-Omni supports DiT-based video models (Sora-style, CogVideo, etc.) when installed with a compatible GPU.

---

Installation

pip install vortexrt

Installs the core engines (Transformers, Diffusers, ONNX Runtime) and all dependencies automatically. For niche engines:

pip install "vortexrt[all]"

---

Quick Start

Text Generation

from vortexrt import VortexRT

vrt = VortexRT()
llm = vrt.load("Qwen/Qwen2.5-1.5B-Instruct")

result = vrt.predict(llm.id, {"prompt": "Explain AI in one sentence."})
print(result.data["generated_text"])
# → "AI is the simulation of human intelligence by machines..."

print(f"Engine: {result.engine}  Time: {result.timing.total_ms:.0f}ms")
print(f"Tokens: {result.usage}")

Chat (Multi-turn)

chat = vrt.chat(llm.id, system="You are a Python expert. Keep replies short.")

reply = chat.send("How do I sort a dict by value?")
print(reply.data["generated_text"])

# Stream token by token
for token in chat.stream("Show me a one-line example"):
    print(token, end="", flush=True)

chat.reset()  # clear history

Image Generation

img = vrt.load("runwayml/stable-diffusion-v1-5")

result = vrt.predict(img.id, {
    "prompt": "A mountain lake at sunrise",
    "num_inference_steps": 20,
    "save_to": True,     # auto-saves to Desktop
    "show_progress": True,  # live progress bar
})
print(result.data["saved_path"])

Multiple Models on One Port

vrt.load("Qwen/Qwen2.5-1.5B-Instruct")
vrt.load("runwayml/stable-diffusion-v1-5")
vrt.serve()  # → http://localhost:8000

Engine Control

# Let the system choose (recommended)
vrt.load("model")

# Force a specific engine
vrt.load("model", engine="transformers")

# With optimizations
vrt.load("model", quantize="int8", torch_dtype="bfloat16")

# See what engine will be chosen
vrt.routing_table()

Authentication

Gated models (Llama, FLUX, Gemma) need a HuggingFace token:

from vortexrt import set_token
set_token("hf_...")

---

REST API

vrt.serve()  # http://localhost:8000

Method	Endpoint	Description
GET	/v1/health	Health + hardware + available engines
GET	/v1/engines	Engine routing table
GET	/v1/models	All loaded models
POST	/v1/models/load	Load a model
POST	/v1/models/{id}/predict	Inference (returns timing + usage)
POST	/v1/models/{id}/stream	SSE token stream
POST	/v1/models/{id}/chat	Multi-turn chat
DELETE	/v1/models/{id}	Unload

---

License

MIT — see LICENSE.

Built on Apache 2.0 and MIT licensed engines. All third-party engines remain under their original licenses.