Plugin for LLM adding fast Cerebras inference API support
Project description
llm plugin to prompt Cerebras hosted models.
Install this plugin in the same environment as LLM:
llm install llm-cerebras
You'll need to obtain a Cerebras API key following the instructions here. Once you have it, configure the plugin like this:
llm keys set cerebras
To use the Cerebras models, run:
llm -m cerebras-llama3.1-8b "Your prompt here"
Or for the 70B model:
llm -m cerebras-llama3.1-70b "Your prompt here"
The following options are available:
temperature
: Controls randomness. Defaults to 0.7, range 0-1.5.max_tokens
: The maximum number of tokens to generate.top_p
: Alternative to temperature for nucleus sampling. Defaults to 1.seed
: For deterministic sampling.
Example usage with options:
llm -m cerebras-llama3.1-8b "Your prompt" -o temperature 0.5 -o max_tokens 100
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd llm-cerebras
python3 -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
llm_cerebras-0.1.3.tar.gz
(3.9 kB
view hashes)
Built Distribution
Close
Hashes for llm_cerebras-0.1.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5eaa9402ff2e4afa7c5a606cc605d5b191a4c9c6d1b64fd3ec2208f72511a8b5 |
|
MD5 | 780508aa321dc919016e0ef9d9a4d50f |
|
BLAKE2b-256 | 43ef1b26f0e2633212f401c38bb4a2aa27e5062638abf658949167f97ba0352e |