llama-index llms gaudi integration
Project description
LlamaIndex Llms Integration with Intel Gaudi
Installation
pip install --upgrade-strategy eager optimum[habana]
pip install llama-index-llms-gaudi
pip install llama-index-llms-huggingface
Usage
import argparse
import os, logging
from llama_index.llms.gaudi import GaudiLLM
def setup_parser(parser):
parser.add_argument(...)
args = parser.parse_args()
return args
if __name__ == "__main__":
parser = argparse.ArgumentParser(
description="GaudiLLM Basic Usage Example"
)
args = setup_parser(parser)
args.model_name_or_path = "HuggingFaceH4/zephyr-7b-alpha"
llm = GaudiLLM(
args=args,
logger=logger,
model_name="HuggingFaceH4/zephyr-7b-alpha",
tokenizer_name="HuggingFaceH4/zephyr-7b-alpha",
query_wrapper_prompt=PromptTemplate(
"<|system|>\n</s>\n<|user|>\n{query_str}</s>\n<|assistant|>\n"
),
context_window=3900,
max_new_tokens=256,
generate_kwargs={"temperature": 0.7, "top_k": 50, "top_p": 0.95},
messages_to_prompt=messages_to_prompt,
device_map="auto",
)
query = "Is the ocean blue?"
print("\n----------------- Complete ------------------")
completion_response = llm.complete(query)
print(completion_response.text)
Examples
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for llama_index_llms_gaudi-0.1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | de2c2a63f343657079624975123e110797d18ae0b6177b0e0d55846f363cae12 |
|
MD5 | fb240ec6827e8aeb31a74d7a757499a2 |
|
BLAKE2b-256 | 089c6ab8a6d0cd99c349c6d0fb511219451ec7e8b37e54f7c80040e0d9859300 |
Close
Hashes for llama_index_llms_gaudi-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d14c303f3380df14357df0ffdf5160ad014e705af62c13552c4a15d27c16ca2a |
|
MD5 | 83ab79649b8ea9358d59a26ef0c6ef32 |
|
BLAKE2b-256 | 893f9dd0bf5f7cd7098fa5b13697c4e7c50735153aa125cddc20ee4529c4185b |