onnxllm 0.0.1

OnnxLLM: A collection of Onnx Large Language Models.

OnnxLLM

This is an example project for learning purpose

Installation

Using Prebuilt

pip install onnxllm

Install From Source

pip install git+https://github.com/inisis/OnnxLLM@main

Install From Local

git clone https://github.com/inisis/OnnxLLM && cd OnnxLLM/
pip install .

How to use

from transformers import AutoTokenizer
from onnxllm import AutoModelForCausalLM

# you should download onnx models from https://huggingface.co/inisis-me first
tokenizer = AutoTokenizer.from_pretrained("/data/llm/llm-export/onnx-standard/", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("/data/llm/llm-export/onnx-standard/", trust_remote_code=True)

prompt = '蒙古国的首都是乌兰巴托（Ulaanbaatar）\n冰岛的首都是雷克雅未克（Reykjavik）\n埃塞俄比亚的首都是'

inputs = tokenizer(prompt, return_tensors='pt')

output = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(output[0], skip_special_tokens=True))

References

• mnn-llm
• onnxruntime
• huggingface

referenced models

• chatglm-6b
• chatglm2-6b
• chatglm3-6b
• codegeex2-6b
• Baichuan2-7B-Chat
• Qwen-7B-Chat
• Qwen-VL-Chat
• Qwen-1.8B-Chat
• Llama-2-7b-chat-ms
• internlm-chat-7b
• phi-2
• bge-large-zh
• TinyLlama-1.1B-Chat-v0.6
• Yi-6B-Chat
• Qwen1.5-0.5B-Chat
• Qwen1.5-1.8B-Chat
• Qwen1.5-4B-Chat
• Qwen1.5-7B-Chat