LlamaIndex Llms Integration: Litellm

Installation

Install the required Python packages:

%pip install llama-index-llms-litellm
!pip install llama-index

Usage

Import Required Libraries

import os
from llama_index.llms.litellm import LiteLLM
from llama_index.core.llms import ChatMessage

Set Up Environment Variables

Set your API keys as environment variables:

os.environ["OPENAI_API_KEY"] = "your-api-key"
os.environ["COHERE_API_KEY"] = "your-api-key"

Example: OpenAI Call

To interact with the OpenAI model:

message = ChatMessage(role="user", content="Hey! how's it going?")
llm = LiteLLM("gpt-3.5-turbo")
chat_response = llm.chat([message])
print(chat_response)

Example: Cohere Call

To interact with the Cohere model:

llm = LiteLLM("command-nightly")
chat_response = llm.chat([message])
print(chat_response)

Example: Chat with System Message

To have a chat with a system role:

messages = [
    ChatMessage(
        role="system", content="You are a pirate with a colorful personality"
    ),
    ChatMessage(role="user", content="Tell me a story"),
]
resp = LiteLLM("gpt-3.5-turbo").chat(messages)
print(resp)

Streaming Responses

To use the streaming feature with stream_complete:

llm = LiteLLM("gpt-3.5-turbo")
resp = llm.stream_complete("Paul Graham is ")
for r in resp:
    print(r.delta, end="")

Streaming Chat Example

To stream chat messages:

llm = LiteLLM("gpt-3.5-turbo")
resp = llm.stream_chat(messages)
for r in resp:
    print(r.delta, end="")

Asynchronous Example

For asynchronous calls, use:

llm = LiteLLM("gpt-3.5-turbo")
resp = await llm.acomplete("Paul Graham is ")
print(resp)

LLM Implementation example

https://docs.llamaindex.ai/en/stable/examples/llm/litellm/

Metadata

Release files for llama-index-llms-litellm 0.8.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for llama-index-llms-litellm 0.8.0
File	Size	Uploaded
llama_index_llms_litellm-0.8.0.tar.gz	10.9 kB	Aug 31, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for llama-index-llms-litellm 0.8.0
File	Interpreter	ABI	Platform	Reset
llama_index_llms_litellm-0.8.0-py3-none-any.whl 11.1 kB Aug 31, 2026	Python 3	none	any	Details

Total release size: 22.1 kB

Release files / llama_index_llms_litellm-0.8.0.tar.gz

Download URL	llama_index_llms_litellm-0.8.0.tar.gz
Size	10.9 kB
Tags	Source
SHA-256 checksum How to use checksums	`8d1d6ee301e43c5cf00aea43b06b3b4d1509fe63693659175921a9f68b5c74db`
BLAKE2b-256 checksum How to use checksums	`737e5cf3a1865adc58b93323a0dcaac5eea50444723c097ad1d438253b54dc36`
Upload date	Aug 31, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.12.7 {"installer":{"name":"uv","version":"0.12.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`

Release files / llama_index_llms_litellm-0.8.0-py3-none-any.whl

Download URL	llama_index_llms_litellm-0.8.0-py3-none-any.whl
Size	11.1 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`b0087bc5bfe3b99f7b1624c2ce4c4a067ebbe09618a1a3d783e4ef384fd6c233`
BLAKE2b-256 checksum How to use checksums	`0b8a5a08690384e148c5289a22d9a08eb1e2bb5f5b3d00ce5249cdb9b4bbb9a1`
Upload date	Aug 31, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.12.7 {"installer":{"name":"uv","version":"0.12.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`