qwerky-vllm-models 0.2.5

vLLM plugin for Qwerky AI MambaInLlama hybrid models

Qwerky vLLM Models

A vLLM plugin for serving Qwerky AI's MambaInLlama hybrid models without the --trust-remote-code flag.

Installation

pip install vllm qwerky-vllm-models

Usage

After installing, serve Qwerky models with vLLM:

vllm serve QwerkyAI/Qwerky-Llama3.2-Mamba-3B-Llama3.3-70B-base-distill --max-model-len 4096

The plugin automatically registers the model architecture with vLLM on import.

Supported Models

• QwerkyAI/Qwerky-Llama3.2-Mamba-3B-Llama3.3-70B-base-distill

How It Works

This package uses vLLM's plugin system (vllm.general_plugins entry point) to register the MambaInLlama model architecture. This means:

• No fork of vLLM required
• No --trust-remote-code flag needed
• Works with standard vLLM installation
• Uses vLLM's native Triton-accelerated Mamba kernels

Requirements

• Python >= 3.10
• vLLM >= 0.14.0
• PyTorch >= 2.0.0

License

Apache 2.0