A simple adapter to use vLLM in your Haystack pipelines.
Project description
vLLM-haystack-adapter
Simply use vLLM in your haystack pipeline, to utilize fast, self-hosted LLMs.
Installation
Install the wrapper via pip: pip install vllm-haystack
Usage
This integration provides two invocation layers:
vLLMInvocationLayer
: To use models hosted on a vLLM server (or any other OpenAI compatible server)vLLMLocalInvocationLayer
: To use locally hosted vLLM models
Use a Model Hosted on a vLLM Server
To utilize the wrapper the vLLMInvocationLayer
has to be used.
Here is a simple example of how a PromptNode
can be created with the wrapper.
from haystack.nodes import PromptNode, PromptModel
from vllm_haystack import vLLMInvocationLayer
model = PromptModel(model_name_or_path="", invocation_layer_class=vLLMInvocationLayer, max_length=256, api_key="EMPTY", model_kwargs={
"api_base" : API, # Replace this with your API-URL
"maximum_context_length": 2048,
})
prompt_node = PromptNode(model_name_or_path=model, top_k=1, max_length=256)
The model will be inferred based on the model served on the vLLM server. For more configuration examples, take a look at the unit-tests.
Hosting a vLLM Server
To create an OpenAI-Compatible Server via vLLM you can follow the steps in the Quickstart section of their documentation.
Use a Model Hosted Locally
⚠️To run vLLM
locally you need to have vllm
installed and a supported GPU.
If you don't want to use an API-Server this wrapper also provides a vLLMLocalInvocationLayer
which executes the vLLM on the same node Haystack is running on.
Here is a simple example of how a PromptNode
can be created with the vLLMLocalInvocationLayer
.
from haystack.nodes import PromptNode, PromptModel
from vllm_haystack import vLLMLocalInvocationLayer
model = PromptModel(model_name_or_path=MODEL, invocation_layer_class=vLLMLocalInvocationLayer, max_length=256, model_kwargs={
"maximum_context_length": 2048,
})
prompt_node = PromptNode(model_name_or_path=model, top_k=1, max_length=256)
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for vllm_haystack-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e1921826da2e435e45b0610b088ce2fd25faf85178803ba233ba6ababb0cdebb |
|
MD5 | ea24db737789f8b6bbc2c4d7a0d032fd |
|
BLAKE2b-256 | 004fe9414e134f1fa2b6091e39547c1d871492a26c727b79c5df37f912683d85 |