No project description provided
Project description
bettershot 💡🚀
⚡️ A Python package for adding error monitoring to LLM Apps in a few minutes ⚡
BetterShot by BerriAI let’s you add error (hallucinations/refusal to answer) monitoring to your LLM App in a few minutes. Like Bugsnag/Sentry for LLM apps!
Getting Started
Install bettershot by running this command.:
pip install bettershot
Using bettershot
It's just 1 line of code:
log(messages=messages, completion=result, user_email="YOUR_EMAIL", query=query)
Here's 2 examples of using it:
Calling the 'raw' OpenAI API
from bettershot import log
import openai
def simple_openai_call(query):
messages = [{'role': 'user', 'content': query}]
completion = openai.ChatCompletion.create(
model="gpt-3.5-turbo",
messages=messages
)
log(messages=messages, completion=completion, user_email="YOUR_EMAIL", query=query) #JUST 1 LINE OF CODE!!
simple_openai_call("hey! how's it going?")
Calling the Langchain OpenAI API
import openai
import langchain
from bettershot import log
def simple_langchain_call(query):
chat = ChatOpenAI(model="gpt-3.5-turbo", temperature=0.7)
prompt = "You are an extremely intelligent AI assistant for BerriAI, answer all questions resepectfully and in a warm tone"
messages = [
SystemMessage(content=prompt),
HumanMessage(content=query)
]
result = chat(messages)
log(messages=messages, completion=completion, user_email="YOUR_EMAIL", query=query) #JUST 1 LINE OF CODE!!
simple_langchain_call("hey! how's it going?")
bettershot automatically evaluates your OpenAI responses to determine if the model either invented new information (hallucination) or refused to answer ("Sorry, as an AI language model...") a user's question.
How does eval work?
Reliable + Fast testing is hard, and that's what we want to tackle.
Each question is evaluated 3 times.
Each evaluation returns either True or False, along with the model's rationale for why it chose what it did.
We pick the evaluation (True/False) that occurs most, along with the model rationale to explain reasoning.
Each question is run in parallel and results are added to your dashboard in real-time.
We will be sharing the prompts soon!
Contributing
We welcome contributions to bettershot! Feel free to create issues/PR's/or DM us (👋 Hi I'm Krrish - +17708783106)
License
bettershot is released under the MIT License.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for bettershot-0.1.91-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0d4587984f2a8241c0dadec2743db248881e18b207af0c917f04ac2d3c3ac8b8 |
|
MD5 | e3075e303089767fb25bc8dc62639011 |
|
BLAKE2b-256 | c03e85af72450463f6a2ca3f5330f87601f258d40de4e2ccf0175c8a33410f5c |