VortexRT is a Python library for AI model inference. Give it a model name, it detects the task, picks the best engine for your hardware, installs missing dependencies, and runs inference.

from vortexrt import VortexRT
vrt = VortexRT()
llm = vrt.load("Qwen/Qwen2.5-1.5B-Instruct")
result = vrt.predict(llm.id, {"prompt": "Explain quantum computing"})

Engines

VortexRT is built on 7 open-source inference engines. Each engine is independently licensed and can be used in any other project.

All engines are open source and permissible for commercial use. Our library is MIT licensed — compatible with all of them.

Hardware Support

Yes, it works on AMD, Intel, and CPU. The library automatically detects your hardware and picks the best available engine. If vLLM isn't available (no NVIDIA GPU), it falls back to Transformers for text and Diffusers for images.

Video Generation

Video generation is registered as a task (text-to-video) but requires a video-capable model and engine. vLLM-Omni supports DiT-based video models (Sora-style, CogVideo, etc.) when installed with a compatible GPU.

Installation

pip install vortexrt

Installs the core engines (Transformers, Diffusers, ONNX Runtime) and all dependencies automatically. For niche engines:

pip install "vortexrt[all]"

Quick Start

Text Generation

from vortexrt import VortexRT

vrt = VortexRT()
llm = vrt.load("Qwen/Qwen2.5-1.5B-Instruct")

result = vrt.predict(llm.id, {"prompt": "Explain AI in one sentence."})
print(result.data["generated_text"])
# → "AI is the simulation of human intelligence by machines..."

print(f"Engine: {result.engine}  Time: {result.timing.total_ms:.0f}ms")
print(f"Tokens: {result.usage}")

Chat (Multi-turn)

chat = vrt.chat(llm.id, system="You are a Python expert. Keep replies short.")

reply = chat.send("How do I sort a dict by value?")
print(reply.data["generated_text"])

# Stream token by token
for token in chat.stream("Show me a one-line example"):
    print(token, end="", flush=True)

chat.reset()  # clear history

Image Generation

img = vrt.load("runwayml/stable-diffusion-v1-5")

result = vrt.predict(img.id, {
    "prompt": "A mountain lake at sunrise",
    "num_inference_steps": 20,
    "save_to": True,     # auto-saves to Desktop
    "show_progress": True,  # live progress bar
})
print(result.data["saved_path"])

Multiple Models on One Port

vrt.load("Qwen/Qwen2.5-1.5B-Instruct")
vrt.load("runwayml/stable-diffusion-v1-5")
vrt.serve()  # → http://localhost:8000

Engine Control

# Let the system choose (recommended)
vrt.load("model")

# Force a specific engine
vrt.load("model", engine="transformers")

# With optimizations
vrt.load("model", quantize="int8", torch_dtype="bfloat16")

# See what engine will be chosen
vrt.routing_table()

Authentication

Gated models (Llama, FLUX, Gemma) need a HuggingFace token:

from vortexrt import set_token
set_token("hf_...")

REST API

vrt.serve()  # http://localhost:8000

License

MIT — see LICENSE.

Built on Apache 2.0 and MIT licensed engines. All third-party engines remain under their original licenses.