Scale LLM Engine Python client
Project description
LLM Engine
The LLM Engine Python library provides a convenient way of interfacing with a
llmengine
endpoint running on
LLM Engine or on your own infrastructure.
Get Started
Install
pip install scale-llm-engine
Usage
If you are using LLM Engine, you can get your API key from
https://spellbook.scale.com/settings.
Set the SCALE_API_KEY
environment variable to your API key.
If you are using your own infrastructure, you can set the
LLM_ENGINE_SERVE_BASE_PATH
environment variable to the base URL of your
self-hosted llmengine
endpoint.
from llmengine import Completion
response = Completion.create(
model="llama-2-7b",
prompt="Hello, my name is",
max_new_tokens=10,
temperature=0.2,
)
print(response.outputs[0].text)
Documentation
Documentation is available at https://scaleapi.github.io/llm-engine/.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
scale_llm_engine-0.0.0b12.tar.gz
(18.5 kB
view hashes)
Built Distribution
Close
Hashes for scale_llm_engine-0.0.0b12.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | b8ae4b7f94a23ff8e948bdaab8fa6cd8a3a1e30519989c9fde3e7a0f5d2a0aa3 |
|
MD5 | 68afb67e12fe7925784865d7f3c96f96 |
|
BLAKE2b-256 | 30abc04ecfd88b12a93149da4537320612457d0c02e1fc1b23cd4645fd46fde7 |
Close
Hashes for scale_llm_engine-0.0.0b12-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6efa7853a95a7c246c97f722b3e73ce0a44b5e46f207ffa6ee1487d6917b884e |
|
MD5 | f96cb120cbfd3b1dec88aa9a3b50374f |
|
BLAKE2b-256 | 2bd091c2095e0c6d9f58aa118c95ddc2d7cbe48763ca2edea7a12afc5d41bc03 |