llm-nosrun

LLM plugin for models hosted by NOS Cloud

Installation

First, install the LLM command-line utility.

Now install this plugin in the same environment as LLM.

llm install llm-nosrun

Configuration

You will need an API key from NOS Cloud. You can obtain one here.

You can set that as an environment variable called LLM_NOSRUN_KEY, or add it to the llm set of saved keys using:

llm keys set nosrun

Enter key: <paste key here>

Usage

To list available models, run:

llm models list

You should see a list that looks something like this:

NOSRun: TinyLlama/TinyLlama-1.1B-Chat-v1.0
NOSRun: meta-llama/Llama-2-7b-chat-hf
NOSRun: meta-llama/Llama-2-13b-chat-hf
NOSRun: meta-llama/Llama-2-70b-chat-hf
NOSRun: HuggingFaceH4/zephyr-7b-beta
NOSRun: HuggingFaceH4/tiny-random-LlamaForCausalLM
NOSRun: NousResearch/Yarn-Mistral-7b-128k
NOSRun: mistralai/Mistral-7B-Instruct-v0.2
NOSRun: mistralai/Mixtral-8x7B-Instruct-v0.1
NOSRun: TheBloke/TinyLlama-1.1B-Chat-v1.0-AWQ
NOSRun: TheBloke/Mixtral-8x7B-Instruct-v0.1-AWQ
NOSRun: mlabonne/phixtral-2x2_8
NOSRun: mlabonne/phixtral-4x2_8

To run a prompt against a model, pass its full model ID to the -m option, like this:

llm -m TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
  'Five strident names for a pet walrus' \
  --system 'You love coming up with creative names for pets'

You can set a shorter alias for a model using the llm aliases command like so:

llm aliases set tinyllama TinyLlama/TinyLlama-1.1B-Chat-v1.0

Now you can prompt Llama 2 70B using:

cat llm_nosrun.py | \
  llm -m tinyllama -s 'explain this code'

Development

To set up this plugin locally, first checkout the code. Then create a new virtual environment:

cd llm-nosrun
python3 -m venv venv
source venv/bin/activate

Now install the dependencies and test dependencies:

pip install -e '.[test]'

To run the tests:

pytest