llama-index llms litellm integration
Project description
LlamaIndex Llms Integration: Litellm
Installation
-
Install the required Python packages:
%pip install llama-index-llms-litellm !pip install llama-index
Usage
Import Required Libraries
import os
from llama_index.llms.litellm import LiteLLM
from llama_index.core.llms import ChatMessage
Set Up Environment Variables
Set your API keys as environment variables:
os.environ["OPENAI_API_KEY"] = "your-api-key"
os.environ["COHERE_API_KEY"] = "your-api-key"
Example: OpenAI Call
To interact with the OpenAI model:
message = ChatMessage(role="user", content="Hey! how's it going?")
llm = LiteLLM("gpt-3.5-turbo")
chat_response = llm.chat([message])
print(chat_response)
Example: Cohere Call
To interact with the Cohere model:
llm = LiteLLM("command-nightly")
chat_response = llm.chat([message])
print(chat_response)
Example: Chat with System Message
To have a chat with a system role:
messages = [
ChatMessage(
role="system", content="You are a pirate with a colorful personality"
),
ChatMessage(role="user", content="Tell me a story"),
]
resp = LiteLLM("gpt-3.5-turbo").chat(messages)
print(resp)
Streaming Responses
To use the streaming feature with stream_complete
:
llm = LiteLLM("gpt-3.5-turbo")
resp = llm.stream_complete("Paul Graham is ")
for r in resp:
print(r.delta, end="")
Streaming Chat Example
To stream chat messages:
llm = LiteLLM("gpt-3.5-turbo")
resp = llm.stream_chat(messages)
for r in resp:
print(r.delta, end="")
Asynchronous Example
For asynchronous calls, use:
llm = LiteLLM("gpt-3.5-turbo")
resp = await llm.acomplete("Paul Graham is ")
print(resp)
LLM Implementation example
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for llama_index_llms_litellm-0.2.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 955f8117170718b627bd48c88213bf9afb9fd70d91fca8cd2455bcdbdf83f3bf |
|
MD5 | 26611fd2a31ba6cd8f4a538c42134fc5 |
|
BLAKE2b-256 | 51930173dc0ef7e5adb62470ba1c6a5b19b77c804e4fac649da5a240cda904e7 |
Close
Hashes for llama_index_llms_litellm-0.2.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 85d53430ad2f7d0697ac55473aaf9628d97db856e40555bf479175f6f31508e4 |
|
MD5 | 100408fbf3e014914d23a295059b9212 |
|
BLAKE2b-256 | 09e5e58e2cdf565e6462b06776ccc5045d57bada3a55e509dccfb9887d535c54 |