Union vLLM Plugin

Serve large language models using vLLM with Flyte Apps.

This plugin provides the VLLMAppEnvironment class for deploying and serving LLMs using vLLM.

Installation

pip install --pre flyteplugins-vllm

Usage

import flyte
import flyte.app
from flyteplugins.vllm import VLLMAppEnvironment

# Define the vLLM app environment
vllm_app = VLLMAppEnvironment(
    name="my-llm-app",
    model_path="s3://your-bucket/models/your-model",
    model_id="your-model-id",
    resources=flyte.Resources(cpu="4", memory="16Gi", gpu="L40s:1"),
    stream_model=True,  # Stream model directly from blob store to GPU
    scaling=flyte.app.Scaling(
        replicas=(0, 1),
        scaledown_after=300,
    ),
)

if __name__ == "__main__":
    flyte.init_from_config()
    app = flyte.serve(vllm_app)
    print(f"Deployed vLLM app: {app.url}")

Features

Streaming Model Loading: Stream model weights directly from object storage to GPU memory, reducing startup time and disk requirements.
Speculative Decoding: Serve a target model alongside a draft model (speculator) to shorten decode.
OpenAI-Compatible API: The deployed app exposes an OpenAI-compatible API for chat completions.
Auto-scaling: Configure scaling policies to scale up/down based on traffic.
Tensor Parallelism: Support for distributed inference across multiple GPUs.

Speculative decoding

Set speculative_config to turn on speculative decoding. It is rendered into vLLM's --speculative-config JSON blob. When the speculator has its own weights, point draft_model_path (object storage, RunOutput or ArtifactValue) or draft_model_hf_path at them and the plugin mounts them next to the target model and fills in the config's model key:

vllm_app = VLLMAppEnvironment(
    name="qwen3-8b-spec",
    model_path="s3://your-bucket/models/qwen3-8b",
    model_id="qwen3-8b",
    draft_model_path="s3://your-bucket/models/qwen3-8b-eagle3",
    speculative_config={"method": "eagle3", "num_speculative_tokens": 3},
    resources=flyte.Resources(cpu="8", memory="64Gi", gpu="L40s:1", disk="120Gi"),
)

Methods that need no separate weights work the same way, without a draft model — n-gram is a useful floor to measure a real speculator against:

speculative_config={"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 4}

Two caveats worth knowing up front:

Streaming is disabled whenever a draft model is configured. The Flyte streaming loader is process-wide and describes a single set of weights, so both models are downloaded to the container instead. Size disk for target + draft + a margin.
Measure acceptance length, not just tokens/sec. A speculator that produces no speedup is indistinguishable from a mis-tuned one until you read vllm:spec_decode_num_accepted_tokens_total / ..._num_drafts_total. Acceptance ≈ 1 means the draft model is wrong for your target.

Extra arguments

extra_args is appended to vllm serve, as either a string or a list:

vllm_app = VLLMAppEnvironment(
    name="my-llm-app",
    model_path="s3://your-bucket/models/your-model",
    model_id="your-model-id",
    extra_args="--max-model-len 8192 --quantization fp8",
)

Arguments are quoted before they reach the server, so values containing spaces or JSON survive intact. Arguments of the form $MY_VAR are left unquoted so that Flyte still expands them from the app's environment.

See the vLLM engine arguments documentation for available options.

Release files for flyteplugins-vllm 2.10.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for flyteplugins-vllm 2.10.1
File	Interpreter	ABI	Platform	Reset
flyteplugins_vllm-2.10.1-py3-none-any.whl 11.8 kB Sep 25, 2026	Python 3	none	any	Details

Release files / flyteplugins_vllm-2.10.1-py3-none-any.whl

Download URL	flyteplugins_vllm-2.10.1-py3-none-any.whl
Size	11.8 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`b38cdd6200e8ef495802429a3742d80e556fb4f65f745fd6bb71625f81b41d96`
BLAKE2b-256 checksum How to use checksums	`cab4659e75dcaca96123471f2cc463a6fe626a474b918ef0093784874dedcae7`
Upload date	Sep 25, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.13.15`