OpenLLM Client: Interacting with OpenLLM HTTP/gRPC server, or any BentoML server.
Project description
📖 Introduction
With OpenLLM, you can run inference with any open-source large-language models, deploy to the cloud or on-premises, and build powerful AI apps, and more.
To learn more about OpenLLM, please visit OpenLLM's README.md
This package holds the underlying client implementation for OpenLLM. If you are
coming from OpenLLM, the client can be accessed via openllm.client
.
It provides somewhat of a "similar" APIs to bentoml.Client
(via openllm_client.min
) for interacting with OpenLLM server. This can also be extended to use with general
BentoML server as well.
[!NOTE] The component of interop with generic BentoML server will be considered as EXPERIMENTAL and will be refactored to new client implementation soon! If you are just using this package for interacting with OpenLLM server, The API should be the same as
openllm.client
namespace.
import openllm
client = openllm.client.HTTPClient()
client.query('Explain to me the difference between "further" and "farther"')
📔 Citation
If you use OpenLLM in your research, we provide a citation to use:
@software{Pham_OpenLLM_Operating_LLMs_2023,
author = {Pham, Aaron and Yang, Chaoyu and Sheng, Sean and Zhao, Shenyang and Lee, Sauyon and Jiang, Bo and Dong, Fog and Guan, Xipeng and Ming, Frost},
license = {Apache-2.0},
month = jun,
title = {{OpenLLM: Operating LLMs in production}},
url = {https://github.com/bentoml/OpenLLM},
year = {2023}
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for openllm_client-0.5.0a1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 074033c6f4c15d5e3ed7378925847b670adde39f51504d77cd980fd6d56b369e |
|
MD5 | 58baf98da89d3d8f511894316435e138 |
|
BLAKE2b-256 | 120b488a932ef0ef95d271d099e445bbe504e2d9a8be7bd3889b3d8d0b3d0ced |