llm-replicate

LLM plugin for models hosted on Replicate

Installation

Install this plugin in the same environment as LLM.

llm install llm-replicate

Configuration

You will need an API key from Replicate. You can obtain one here.

You can set that as an environment variable called REPLICATE_API_TOKEN, or add it to the llm set of saved keys using:

llm keys set replicate

Enter key: <paste key here>

To fetch and save details of the default collection of language models hosted on Replicate, run this:

llm replicate fetch-models

Usage

Run llm models list to see the list of models:

llm models list

Then run a prompt through a specific model like this:

llm -m replicate-llama-7b "Ten great names for a pet pelican"

Registering extra models

To register additional models that are not included in the default Language models collection, find their ID on Replicate and use the llm replicate add command.

For example, to add the joehoover/falcon-40b-instruct model, run this:

llm replicate add joehoover/falcon-40b-instruct --alias falcon

This adds the model with the alias falcon - you can have 0 or more aliases for a model.

Now you can run it like this:

llm -m replicate-joehoover-falcon-40b-instruct "Three reasons to get a pet falcon"

Or using the alias like this:

llm -m falcon "Three reasons to get a pet falcon"

You can edit the list of models you have registered using the default $EDITOR like this:

llm replicate edit-models

Development

To set up this plugin locally, first checkout the code. Then create a new virtual environment:

cd llm-palm
python3 -m venv venv
source venv/bin/activate

Now install the dependencies and test dependencies:

pip install -e '.[test]'

To run the tests:

pytest