llama-index llms deepinfra integration
Project description
LlamaIndex Llms Integration: DeepInfra
Installation
First, install the necessary package:
pip install llama-index-llms-deepinfra
Initialization
Set up the DeepInfraLLM
class with your API key and desired parameters:
from llama_index.llms.deepinfra import DeepInfraLLM
import asyncio
llm = DeepInfraLLM(
model="mistralai/Mixtral-8x22B-Instruct-v0.1", # Default model name
api_key="your-deepinfra-api-key", # Replace with your DeepInfra API key
temperature=0.5,
max_tokens=50,
additional_kwargs={"top_p": 0.9},
)
Synchronous Complete
Generate a text completion synchronously using the complete
method:
response = llm.complete("Hello World!")
print(response.text)
Synchronous Stream Complete
Generate a streaming text completion synchronously using the stream_complete
method:
content = ""
for completion in llm.stream_complete("Once upon a time"):
content += completion.delta
print(completion.delta, end="")
Synchronous Chat
Generate a chat response synchronously using the chat
method:
from llama_index.core.base.llms.types import ChatMessage
messages = [
ChatMessage(role="user", content="Tell me a joke."),
]
chat_response = llm.chat(messages)
print(chat_response.message.content)
Synchronous Stream Chat
Generate a streaming chat response synchronously using the stream_chat
method:
messages = [
ChatMessage(role="system", content="You are a helpful assistant."),
ChatMessage(role="user", content="Tell me a story."),
]
content = ""
for chat_response in llm.stream_chat(messages):
content += chat_response.message.delta
print(chat_response.message.delta, end="")
Asynchronous Complete
Generate a text completion asynchronously using the acomplete
method:
async def async_complete():
response = await llm.acomplete("Hello Async World!")
print(response.text)
asyncio.run(async_complete())
Asynchronous Stream Complete
Generate a streaming text completion asynchronously using the astream_complete
method:
async def async_stream_complete():
content = ""
response = await llm.astream_complete("Once upon an async time")
async for completion in response:
content += completion.delta
print(completion.delta, end="")
asyncio.run(async_stream_complete())
Asynchronous Chat
Generate a chat response asynchronously using the achat
method:
async def async_chat():
messages = [
ChatMessage(role="user", content="Tell me an async joke."),
]
chat_response = await llm.achat(messages)
print(chat_response.message.content)
asyncio.run(async_chat())
Asynchronous Stream Chat
Generate a streaming chat response asynchronously using the astream_chat
method:
async def async_stream_chat():
messages = [
ChatMessage(role="system", content="You are a helpful assistant."),
ChatMessage(role="user", content="Tell me an async story."),
]
content = ""
response = await llm.astream_chat(messages)
async for chat_response in response:
content += chat_response.message.delta
print(chat_response.message.delta, end="")
asyncio.run(async_stream_chat())
For any questions or feedback, please contact us at feedback@deepinfra.com.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for llama_index_llms_deepinfra-0.1.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | c6fc578e0857999f8b31d2c14c252f43ecb07b382a97e0486a7d77dc35665296 |
|
MD5 | 8e71b92b8b3e5d881ce44cf6af2102a3 |
|
BLAKE2b-256 | 2789d8443e22e440f6dd53a66bf5ecaac981e034631369fe347a3e37851858ba |
Hashes for llama_index_llms_deepinfra-0.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3c75f2e0f89baa09101b2e8bd901b68557fab9538f9a800883334f3e73c5f5a0 |
|
MD5 | 2ad4c36bf749f01b49d88d159b62c8c5 |
|
BLAKE2b-256 | bb15751aac624565627b16d0ca33857e6830c2a5d80d1aa124b17e231323643a |