Modelz LLM is an inference server that facilitates the utilization of open source large language models (LLMs), such as FastChat, LLaMA, and ChatGLM, on either local or cloud-based environments with OpenAI compatible API.
Features
- OpenAI compatible API: Modelz LLM provides an OpenAI compatible API for LLMs, which means you can use the OpenAI python SDK or LangChain to interact with the model.
- Self-hosted: Modelz LLM can be easily deployed on either local or cloud-based environments.
- Open source LLMs: Modelz LLM supports open source LLMs, such as FastChat, LLaMA, and ChatGLM.
- Cloud native: We provide docker images for different LLMs, which can be easily deployed on Kubernetes, or other cloud-based environments (e.g. Modelz)
Quick Start
Install
pip install modelz-llm
# or install from source
pip install git+https://github.com/tensorchord/modelz-llm.git[gpu]
Run the self-hosted API server
Please first start the self-hosted API server by following the instructions:
modelz-llm -m bigscience/bloomz-560m --device cpu
Currently, we support the following models:
| Model Name | Huggingface Model | Docker Image | Recommended GPU |
|---|---|---|---|
| FastChat T5 | lmsys/fastchat-t5-3b-v1.0 |
modelzai/llm-fastchat-t5-3b | Nvidia L4(24GB) |
| Vicuna 7B Delta V1.1 | lmsys/vicuna-7b-delta-v1.1 |
modelzai/llm-vicuna-7b | Nvidia A100(40GB) |
| LLaMA 7B | decapoda-research/llama-7b-hf |
modelzai/llm-llama-7b | Nvidia A100(40GB) |
| ChatGLM 6B INT4 | THUDM/chatglm-6b-int4 |
modelzai/llm-chatglm-6b-int4 | Nvidia T4(16GB) |
| ChatGLM 6B | THUDM/chatglm-6b |
modelzai/llm-chatglm-6b | Nvidia L4(24GB) |
| Bloomz 560M | bigscience/bloomz-560m |
modelzai/llm-bloomz-560m | CPU |
| Bloomz 1.7B | bigscience/bloomz-1b7 |
CPU | |
| Bloomz 3B | bigscience/bloomz-3b |
Nvidia L4(24GB) | |
| Bloomz 7.1B | bigscience/bloomz-7b1 |
Nvidia A100(40GB) |
Use OpenAI python SDK
Then you can use the OpenAI python SDK to interact with the model:
import openai
openai.api_base="http://localhost:8000"
openai.api_key="any"
# create a chat completion
chat_completion = openai.ChatCompletion.create(model="any", messages=[{"role": "user", "content": "Hello world"}])
Integrate with Langchain
You could also integrate modelz-llm with langchain:
import openai
openai.api_base="http://localhost:8000"
openai.api_key="any"
from langchain.llms import OpenAI
llm = OpenAI()
llm.generate(prompts=["Could you please recommend some movies?"])
Deploy on Modelz
You could also deploy the modelz-llm directly on Modelz:
Supported APIs
Modelz LLM supports the following APIs for interacting with open source large language models:
/completions/chat/completions/embeddings/engines/<any>/embeddings/v1/completions/v1/chat/completions/v1/embeddings
Acknowledgements
Metadata
Release files for modelz-llm 23.7.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| modelz-llm-23.7.4.tar.gz | 21.2 kB | Without provenance | Details for modelz-llm-23.7.4.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| modelz_llm-23.7.4-py3-none-any.whl | Python 3 | none | any | Details for modelz_llm-23.7.4-py3-none-any.whl |
Total release size: 33.9 kB
Release files / modelz-llm-23.7.4.tar.gz
| Download URL | modelz-llm-23.7.4.tar.gz |
|---|---|
| Size | 21.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3a5a7fcde99f82a9d6c208fcb4bac683030cf7d5c14c9caad3507cf2554226f2
|
|
BLAKE2b-256 checksum How to use checksums |
121cb914663cf5932e2439b5af2caae24524c077b9e17aea01d9c23f9a99bc93
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.4
|
Release files / modelz_llm-23.7.4-py3-none-any.whl
| Download URL | modelz_llm-23.7.4-py3-none-any.whl |
|---|---|
| Size | 12.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9e7ca6182421a49ba7e43309af0991426ab82a5f6a42e28773d10ed0f7369d1d
|
|
BLAKE2b-256 checksum How to use checksums |
188b8ed002926c89fc211b65f682d5e56fa22089a2ca06e19a3586ad803bd5e5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.4
|
Security for modelz-llm 23.7.4
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.