A simple adapter to use vLLM in your Haystack pipelines.
Project description
vLLM-haystack-adapter
Simply use vLLM in your haystack pipeline, to utilize fast, self-hosted LLMs.
Installation
Install the wrapper via pip: pip install vllm-haystack
Usage
To utilize the wrapper the vLLMInvocationLayer
has to be used.
Here is a simple example of how a PromptNode
can be created with the wrapper.
from haystack.nodes import PromptNode, PromptModel
from vllm_haystack import vLLMInvocationLayer
model = PromptModel(model_name_or_path="", invocation_layer_class=vLLMInvocationLayer, max_length=256, api_key="EMPTY", model_kwargs={
"api_base" : API, # Replace this with your API-URL
"maximum_context_length": 2048,
})
prompt_node = PromptNode(model_name_or_path=model, top_k=1, max_length=256)
For more configuration examples, take a look at the unit-tests.
Hosting a vLLM Server
To create an OpenAI-Compatible Server via vLLM you can follow the steps in the Quickstart section of their documentation.
Running Locally
If you don't want to use an API-Server this wrapper also provides a vLLMLocalInvocationLayer
which executes the vLLM on the same node Haystack is running on.
Here is a simple example of how a PromptNode
can be created with the vLLMLocalInvocationLayer
.
from haystack.nodes import PromptNode, PromptModel
from vllm_haystack import vLLMLocalInvocationLayer
model = PromptModel(model_name_or_path=MODEL, invocation_layer_class=vLLMLocalInvocationLayer, max_length=256, model_kwargs={
"maximum_context_length": 2048,
})
prompt_node = PromptNode(model_name_or_path=model, top_k=1, max_length=256)
⚠️To run vLLM
locally yopu need to have vLLM
installed and a supported GPU.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for vllm_haystack-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e0a0046e8c7bef4ab0f976aaea39002a98a497e60b251a9221de9768b30c993e |
|
MD5 | 495f4886af0ec1ca9f864f6282ec4714 |
|
BLAKE2b-256 | cbccbbd676bed2e8c46b79ea857e9648061e8abbcc7dc3e398624adba4c8d3c1 |