llama-index llms openrouter integration
Project description
LlamaIndex Llms Integration: Openrouter
Installation
To install the required packages, run:
%pip install llama-index-llms-openrouter
!pip install llama-index
Setup
Initialize OpenRouter
You need to set either the environment variable OPENROUTER_API_KEY
or pass your API key directly in the class constructor. Replace <your-api-key>
with your actual API key:
from llama_index.llms.openrouter import OpenRouter
from llama_index.core.llms import ChatMessage
llm = OpenRouter(
api_key="<your-api-key>",
max_tokens=256,
context_window=4096,
model="gryphe/mythomax-l2-13b",
)
Generate Chat Responses
You can generate a chat response by sending a list of ChatMessage
instances:
message = ChatMessage(role="user", content="Tell me a joke")
resp = llm.chat([message])
print(resp)
Streaming Responses
To stream responses, use the stream_chat
method:
message = ChatMessage(role="user", content="Tell me a story in 250 words")
resp = llm.stream_chat([message])
for r in resp:
print(r.delta, end="")
Complete with Prompt
You can also generate completions with a prompt using the complete
method:
resp = llm.complete("Tell me a joke")
print(resp)
Streaming Completion
To stream completions, use the stream_complete
method:
resp = llm.stream_complete("Tell me a story in 250 words")
for r in resp:
print(r.delta, end="")
Model Configuration
To use a specific model, you can specify it during initialization. For example, to use Mistral's Mixtral model, you can set it like this:
llm = OpenRouter(model="mistralai/mixtral-8x7b-instruct")
resp = llm.complete("Write a story about a dragon who can code in Rust")
print(resp)
LLM Implementation example
https://docs.llamaindex.ai/en/stable/examples/llm/openrouter/
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for llama_index_llms_openrouter-0.2.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | a45f03d34ef03e774f2e501c4200d2bd22e80504ef2c210783d953166a6973eb |
|
MD5 | 2ad39ce98ec35445c9b5fae316f6f8e3 |
|
BLAKE2b-256 | 6dad7dcc6d39a3d08bc815bdcf62e40456332a004e071ac0ea88ee6d9b2023b4 |
Hashes for llama_index_llms_openrouter-0.2.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | da47f3c2bf985680e65cd0b77e759b1db6568654d4fb8dbd6dc16ac7d6a548c9 |
|
MD5 | 2e6e73a92f92068d336dafa99141fe92 |
|
BLAKE2b-256 | 961cbf8a8531dbfe3493601cb20f8570eea6e04c5abb27869ca9fb0368570e67 |