qwerky-vllm-models 0.2.9

vLLM plugin for Qwerky AI MambaInLlama hybrid models

Qwerky vLLM Models

A vLLM plugin for serving Qwerky AI's MambaInLlama hybrid models without the --trust-remote-code flag.

Installation

pip install vllm qwerky-vllm-models

Usage

After installing, serve Qwerky models with vLLM:

vllm serve QwerkyAI/Qwerky-Llama3.2-Mamba-3B-Llama3.3-70B-base-distill --max-model-len 4096

The plugin automatically registers the model architecture with vLLM on import.

Supported Models

• QwerkyAI/Qwerky-Llama3.2-Mamba-3B-Llama3.3-70B-base-distill

How It Works

This package uses vLLM's plugin system (vllm.general_plugins entry point) to register the MambaInLlama model architecture. This means:

• No fork of vLLM required
• No --trust-remote-code flag needed
• Works with standard vLLM installation
• Uses vLLM's native Triton-accelerated Mamba kernels

Requirements

• Python >= 3.10
• vLLM >= 0.14.0
• PyTorch >= 2.0.0

Changelog

0.2.9

• Fixed weight loading: split fused mha.in_proj into separate q/k/v projections
• Renamed mha.out_proj to o_proj for checkpoint compatibility
• Should now load all ~395 parameters instead of just 163

0.2.8

• Fixed dtype mismatch in SSM scan: F.softplus/torch.exp compute in float32, now cast back to original dtype
• This caused "expected BFloat16 but found Float" error in einsum

0.2.7

• Fixed tensor broadcasting bug in _ssm_scan: A.unsqueeze(0).unsqueeze(-1) -> A.unsqueeze(0).unsqueeze(2)
• This caused shape mismatch (8192 vs 16) during SSM discretization

0.2.6

• Added embed_input_ids method required by vLLM's VllmModelForTextGeneration interface
• This was the root cause of "This model does not support --runner generate" error

0.2.5

• Fixed vLLM runner detection: added MambaInLlamaMambaForCausalLM alias for HF config compatibility
• Added proper protocol inheritance (HasInnerState, IsHybrid) from vllm.model_executor.models.interfaces
• Fixed class variable type hints (ClassVar[Literal[True]]) for vLLM model inspection
• Simplified model registration code

0.2.4

• Complete architecture rewrite with explicit state cache management
• Separate prefill and decode paths for Mamba layers
• Grouped-head Mamba support (num_xb_head, num_C_head, repeat_group)
• Pure PyTorch SSM implementation (preparing for vLLM Triton op integration)

0.2.3

• Fixed d_xb default value computation in configuration
• Removed unsupported device/dtype kwargs from RMSNorm calls

0.2.2

• Fixed vLLM 0.14+ compatibility issues with Mamba ops API

0.2.1

• Updated README, removed SFT model reference

0.2.0

• Initial public release with vLLM plugin system integration

License

Apache 2.0