tools for testing, debugging, and evaluating llm features.
Project description
Baserun
Baserun is the testing and observability platform for LLM apps.
Quick Start
1. Install Baserun
pip install baserun==1.0.0b6
2. Set the Baserun API key
Create an account at https://baserun.ai. Then generate an API key for your project in the settings tab. Set it as an environment variable:
export BASERUN_API_KEY="your_api_key_here"
Usage
In order to have Baserun trace your LLM Requests, all you need to do is import OpenAI
from baserun
instead of openai
. Creating an OpenAI client object automatically starts the trace, and all future LLM requests made with this client object will be captured.
from baserun import OpenAI
def example():
client = OpenAI()
completion = client.chat.completions.create(
name="Paris Activities",
model="gpt-4o",
temperature=0.7,
messages=[
{
"role": "user",
"content": "What are three activities to do in Paris?"
}
],
)
if __name__ == "__main__":
print(example())
Alternate init method
If, for some reason, you don't wish to use Baserun's OpenAI client, you can simply wrap your normal OpenAI client using init
.
from baserun import init
client = init(OpenAI())
Configuring the trace
When you start a trace by initializing an OpenAI object, there are several optional parameters you can set for that trace:
name
: A customized name for the traceresult
: Some end result or output for the traceuser
: A username or user ID to associate with this trace.session
: A session ID to associate with this trace.trace_id
: A previously-generated or custom UUID (e.g. to continue a previous trace)
from baserun import OpenAI
def example():
client = OpenAI(result="What are three activities to do in Paris?")
client.name = "Example"
client.user = "user123"
client.session = "session123"
completion = client.chat.completions.create(
name="Paris Activities",
model="gpt-4o",
temperature=0.7,
messages=[
{
"role": "user",
"content": "What are three activities to do in Paris?"
}
],
)
client.result = "Done"
Evals
You can perform evals directly on a completion object. The includes
eval is used here as an example, and checks if a string is included in the completion's output. The argument passed to eval()
is a name or label used for your reference.
from baserun import OpenAI
def example():
client = OpenAI()
completion = client.chat.completions.create(
model="gpt-4o",
temperature=0.7,
messages=[
{
"role": "user",
"content": "What are three activities to do in Paris?"
}
],
)
client.eval("include_eiffel_tower").includes("Eiffel Tower")
Tags
You can add tags either to the traced OpenAI object or to the completion. There are several different types of tags:
log
: Any arbitrary logs you want to attach to a trace or completionfeedback
: Any score-based feedback given from users (e.g. thumbs up/down, star rating)variable
: Any variables used, e.g. while rendering a templateannotate
: Any arbitrary attributes you want to attach to a trace or completion
Each tag type has functions on traced OpenAI objects and completions. Each tag function can accept a metadata
parameter which is an arbitrary dictionary with any values you might want to capture.
from baserun import OpenAI
def example():
client = OpenAI()
client.log("Gathering user input")
city = input()
completion = client.chat.completions.create(
model="gpt-4o",
temperature=0.7,
messages=[
{
"role": "user",
"content": f"What are three activities to do in {city}?"
}
],
)
completion.variable("city", city)
user_score = input()
client.feedback("User Score", score=user_score, metadata={"My key": "My value"})
Further Documentation
For a deeper dive on all capabilities and more advanced usage, please refer to our Documentation.
License
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for baserun-1.0.0b6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 59946fd9d3ca71e926a6edb0b0b11750da2e30c5e1686a32f73263f9fad5c6e8 |
|
MD5 | 13a9c9ab7f591055edf5eb8324694f20 |
|
BLAKE2b-256 | e7d3ce88927d4100c1671799aaa42e3aa10b9e6bd0250173505f8902156ab6d8 |