llama-index-llms-text-generation-inference 0.3.3

llama-index llms huggingface text generation inference integration

LlamaIndex Llms Integration: Text Generation Inference

⚠️ This integration has been deprecated!

The TextGenerationInference is no longer maintained. Instead, you can use HuggingFaceInferenceAPI. The underlying Text Generation Inference SDK (tgi) has been deprecated in favor of huggingface_hub, which HuggingFaceInferenceAPI is built on top of.

Instead, use llama-index-llms-huggingface-api:

pip install llama-index-llms-huggingface-api

Usage:

from llama_index.llms.huggingface_api import HuggingFaceInferenceAPI

# access hugging face inference
hub_llm = HuggingFaceInferenceAPI(model="openai-community/gpt2")
# or with a local TGI server
tgi_llm = HuggingFaceInferenceAPI(model="http://localhost:8080")