vortexrt 0.3.0

Unified AI model inference — auto-detect, auto-route, auto-serve with multi-engine priority routing.

VortexRT

Unified AI Model Inference — One Library, Every Model

---

VortexRT is a Python library for AI model inference. Give it a model name, it detects the task, picks the best engine for your hardware, installs missing dependencies, and runs inference.

from vortexrt import VortexRT
vrt = VortexRT()
llm = vrt.load("Qwen/Qwen2.5-1.5B-Instruct")
result = vrt.predict(llm.id, {"prompt": "Explain quantum computing"})

---

Engines

VortexRT is built on 7 open-source inference engines. Each engine is independently licensed and can be used in any other project.

Engine	License	Specialisation	% of Tasks
Transformers	Apache 2.0	Text, classification, QA, NER, embeddings, TTS, ASR	45%
Diffusers	Apache 2.0	Text-to-image, image-to-image	10%
ONNX Runtime	MIT	Custom ONNX, audio, TTS, ASR, DirectML	15%
vLLM	Apache 2.0	High-throughput LLM (PagedAttention)	10%
vLLM-Omni	Apache 2.0	Accelerated image/video (DiT, Cache-DiT)	10%
llama.cpp	MIT	GGUF LLMs on CPU	5%
Stable Diffusion.cpp	MIT	GGUF diffusion on CPU/GPU	5%

All engines are open source and permissible for commercial use. Our library is MIT licensed — compatible with all of them.

Hardware Support

	NVIDIA CUDA	AMD ROCm	Intel XPU/Arc	CPU Only	Apple MPS
Transformers	✓	✓	✓	✓	✓
Diffusers	✓	—	—	✓	✓
ONNX Runtime	✓ CUDA EP	✓ ROCm EP	✓ OpenVINO EP	✓	✓ CoreML
vLLM	✓	✓	✓	✓ (slow)	—
vLLM-Omni	✓	✓	✓	—	—
llama.cpp	✓ Vulkan	✓ Vulkan	✓ Vulkan	✓	✓ Metal
diffusion-gguf	✓ Vulkan	✓ Vulkan	✓ Vulkan	✓	—

Yes, it works on AMD, Intel, and CPU. The library automatically detects your hardware and picks the best available engine. If vLLM isn't available (no NVIDIA GPU), it falls back to Transformers for text and Diffusers for images.

Video Generation

Video generation is registered as a task (text-to-video) but requires a video-capable model and engine. vLLM-Omni supports DiT-based video models (Sora-style, CogVideo, etc.) when installed with a compatible GPU.

---

Installation

pip install vortexrt

Installs the core engines (Transformers, Diffusers, ONNX Runtime) and all dependencies automatically. For niche engines:

pip install "vortexrt[all]"

---

Quick Start

Text Generation

from vortexrt import VortexRT

vrt = VortexRT()
llm = vrt.load("Qwen/Qwen2.5-1.5B-Instruct")

result = vrt.predict(llm.id, {"prompt": "Explain AI in one sentence."})
print(result.data["generated_text"])
# → "AI is the simulation of human intelligence by machines..."

print(f"Engine: {result.engine}  Time: {result.timing.total_ms:.0f}ms")
print(f"Tokens: {result.usage}")

Chat (Multi-turn)

chat = vrt.chat(llm.id, system="You are a Python expert. Keep replies short.")

reply = chat.send("How do I sort a dict by value?")
print(reply.data["generated_text"])

# Stream token by token
for token in chat.stream("Show me a one-line example"):
    print(token, end="", flush=True)

chat.reset()  # clear history

Image Generation

img = vrt.load("runwayml/stable-diffusion-v1-5")

result = vrt.predict(img.id, {
    "prompt": "A mountain lake at sunrise",
    "num_inference_steps": 20,
    "save_to": True,     # auto-saves to Desktop
    "show_progress": True,  # live progress bar
})
print(result.data["saved_path"])

Multiple Models on One Port

vrt.load("Qwen/Qwen2.5-1.5B-Instruct")
vrt.load("runwayml/stable-diffusion-v1-5")
vrt.serve()  # → http://localhost:8000

Engine Control

# Let the system choose (recommended)
vrt.load("model")

# Force a specific engine
vrt.load("model", engine="transformers")

# With optimizations
vrt.load("model", quantize="int8", torch_dtype="bfloat16")

# See what engine will be chosen
vrt.routing_table()

Authentication

Gated models (Llama, FLUX, Gemma) need a HuggingFace token:

from vortexrt import set_token
set_token("hf_...")

---

REST API

vrt.serve()  # http://localhost:8000

Method	Endpoint	Description
GET	/v1/health	Health + hardware + available engines
GET	/v1/engines	Engine routing table
GET	/v1/models	All loaded models
POST	/v1/models/load	Load a model
POST	/v1/models/{id}/predict	Inference (returns timing + usage)
POST	/v1/models/{id}/stream	SSE token stream
POST	/v1/models/{id}/chat	Multi-turn chat
DELETE	/v1/models/{id}	Unload

---

Deployment

1. Build the Package

pip install build
cd VortexRT
python -m build

This creates dist/vortexrt-0.3.0.tar.gz and dist/vortexrt-0.3.0-py3-none-any.whl.

2. Test on Test PyPI (Optional)

pip install twine
twine upload --repository testpypi dist/*
pip install --index-url https://test.pypi.org/simple/ vortexrt

3. Publish to PyPI

twine upload dist/*

Enter your PyPI token from https://pypi.org/manage/account/token/.

4. Verify

pip install vortexrt
python -c "from vortexrt import VortexRT; print('OK')"

5. For Each New Version

Update version in two files:

• pyproject.toml → version = "0.4.0"
• vortexrt/__init__.py → __version__ = "0.4.0"

Rebuild and upload. Existing users get an automatic update notification.

6. Installing from Source (Users)

git clone https://github.com/vortexrt/vortexrt
cd VortexRT
pip install -e .

---

API Reference

VortexRT

Method	Returns	Description
load(model_id, *, task?, device?, engine?, quantize?, attn_implementation?, torch_dtype?, **kw)	ModelHandle	Load a model
unload(handle_id)	—	Free resources
predict(handle_id, inputs)	PredictResult	Run inference
stream(handle_id, inputs)	Iterator[dict]	Token-by-token stream
chat(handle_id, *, system?, **kw)	ChatSession	Multi-turn conversation
schema(handle_id)	ModelSchema	I/O contract
models()	List[ModelHandle]	All loaded models
hardware()	dict	CPU, RAM, GPU snapshot
routing_table()	List[dict]	Engine routing with scores
available_engines()	List[str]	Names of available engines
recommend_models(category?)	List[dict]	Curated model catalogue
serve(blocking?)	—	Start REST API
shutdown()	—	Unload all

ChatSession

Method	Description
send(message, **overrides) → PredictResult	Send message, get reply
stream(message, **overrides) → Iterator[str]	Send message, stream reply
reset()	Clear history, keep system prompt
fork() → ChatSession	Independent copy
messages → List[Message]	Full conversation history

PredictResult

Field	Type	Description
data	dict	Inference output
engine	str	Engine that served the request
timing.total_ms	float	Wall-clock duration
timing.engine_ms	float	Engine execution time
usage	dict or None	prompt_tokens, completion_tokens, total_tokens
schema	ModelSchema	Model I/O contract

---

License

MIT — see LICENSE.

Built on Apache 2.0 and MIT licensed engines. All third-party engines remain under their original licenses.