🐍 | Python library for langchain using RunPod API endpoint as LLM.
Project description
Runpod LLM API Endpoint Lib for LangChain
Installation
# Install the latest release version
pip install runpod-llm
# or
# Install the latest development version (main branch)
pip install git+https://https://github.com/tsangwailam/langchain-runpod-llm
Get Runpod API key
- Goto www.runpod.io. Create a RunPod account.
- From the portal, goto Settings>APIKeys
- Create a new API key by click the "+ API Key" button.
Usage
from runpod_llm import RunpodLlama2
llm = RunpodLlama2(
apikey="YOU_RUNPOD_API_KEY",
llm_type="7b|13b",
config={
"max_tokens": 500,
#Maximum number of tokens to generate per output sequence.
"n": 1, # Number of output sequences to return for the given prompt.
"best_of": 1, # Number of output sequences that are generated from the prompt. From these best_of sequences, the top n sequences are returned. best_of must be greater than or equal to n. This is treated as the beam width when use_beam_search is True. By default, best_of is set to n.
"Presence penalty": 0.2, # Float that penalizes new tokens based on whether they appear in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens.
"Frequency penalty": 0.5, # Float that penalizes new tokens based on their frequency in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens.
"temperature": 0.3, # Float that controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. Zero means greedy sampling.
"top_p": 1, # Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens.
"top_k": -1, # Integer that controls the number of top tokens to consider. Set to -1 to consider all tokens.
"use_beam_search": False, # Whether to use beam search instead of sampling.
},
verbose=True, # verbose output
)
some_prompt_template = xxxxx
output_chain = some_prompt_template | llm
output_chain.invoke({"input":"some input to prompt template"})
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for langchain-runpod-llm-0.0.4.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 695f8291a46d3c23640c6a3f96b52f7a8537a196e9309b281dca2595f59f0b33 |
|
MD5 | 5c490fd6c323410757e3fc25ed0ba17b |
|
BLAKE2b-256 | f985db36c7dda297f34ae96d7ed45f4866a21e0340aef94d9ca0b1f53557ba6e |
Close
Hashes for langchain_runpod_llm-0.0.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f890ea51ab0feb88c2a1c29ad476c6ac6df729aba3d882daad251cde0b4d8750 |
|
MD5 | 68f839f51e38fa4fc2bf3387eaef8eaa |
|
BLAKE2b-256 | 48d05e61a6cfd27666880b5cf760e459dd8e12a84917ef0156d4e7acb62d2315 |