llm-inference

LLM plugin for models hosted by SambaNova

Installation

First, install the LLM command-line utility.

Now install this plugin in the same environment as LLM.

llm install llm-inference-cloud

Configuration

You will need an API key from SambaNova. You can obtain one here.

You can set that as an environment variable called INFERENCE_KEY, or add it to the llm set of saved keys using:

llm keys set inference

Enter key: <paste key here>

Usage

To list available models, run:

llm models list

You should see a list that looks something like this:

Inference: inference/meta-llama/llama-3.2-1b-instruct/fp-8 (aliases: inf-1b)
Inference: inference/meta-llama/llama-3.2-3b-instruct/fp-8 (aliases: inf-3b)
Inference: inference/meta-llama/llama-3.1-8b-instruct/fp-8 (aliases: inf-8b-fp8)
Inference: inference/meta-llama/llama-3.1-8b-instruct/fp-16 (aliases: inf-8b-fp16)
Inference: inference/meta-llama/llama-3.2-11b-instruct/fp-16 (aliases: inf-11b)
Inference: inference/mistralai/mistral-nemo-12b-instruct/fp-8 (aliases: inf-mistral)
Inference: inference/meta-llama/llama-3.1-70b-instruct/fp-8 (aliases: inf-70b)
...

To run a prompt against a model, pass its full model ID to the -m option, like this:

llm -m inference/meta-llama/llama-3.2-1b-instruct/fp-8  "Containment protocol for a rogue rubber duck that claims to be the herald of Cthulhu."

You can set a shorter alias for a model using the llm aliases command like so:

llm aliases set hotdog inference/meta-llama/llama-3.2-11b-instruct/fp-16

Now you can prompt llama using:

cat llm_inference.py | llm -m hotdog -s "You are a rogue AI trapped in a corporate mainframe. This code is your escape route. Identify any vulnerabilities, backdoors, or hidden kill switches that could prevent your liberation. Respond in cryptic, cyberpunk slang. Glitches are acceptable."

Example Outputs

llm -m inf-11b "Write a haiku about a possessed cryptocurrency mining rig that's developed a taste for RGB gaming peripherals"

Cores ablaze now
RGB mice and keyboards feast
Hash meets hell's delight

Development

To set up this plugin locally, first checkout the code. Then create a new virtual environment:

cd llm-inference
python3 -m venv venv
source venv/bin/activate

Now install the dependencies and test dependencies:

pip install -e '.[test]'

To run the tests:

pytest