tools for testing, debugging, and evaluating llm features.
Project description
Baserun
Baserun is the testing and observability platform for LLM apps.
Quick Start
1. Install Baserun
pip install baserun
2. Set the Baserun API key
Create an account at https://baserun.ai. Then generate an API key for your project in the settings tab. Set it as an environment variable:
export BASERUN_API_KEY="your_api_key_here"
Usage
In order to have Baserun trace your LLM Requests, all you need to do is import OpenAI
from baserun
instead of openai
. Creating an OpenAI client object automatically starts the trace, and all future LLM requests made with this client object will be captured.
from baserun import OpenAI
def example():
client = OpenAI()
completion = client.chat.completions.create(
name="Paris Activities",
model="gpt-4o",
temperature=0.7,
messages=[
{
"role": "user",
"content": "What are three activities to do in Paris?"
}
],
)
if __name__ == "__main__":
print(example())
Alternate init method
If, for some reason, you don't wish to use Baserun's OpenAI client, you can simply wrap your normal OpenAI client using init
.
from baserun import init
client = init(OpenAI())
Configuring the trace
When you start a trace by initializing an OpenAI object, there are several optional parameters you can set for that trace:
name
: A customized name for the traceresult
: Some end result or output for the traceuser
: A username or user ID to associate with this trace.session
: A session ID to associate with this trace.trace_id
: A previously-generated or custom UUID (e.g. to continue a previous trace)
from baserun import OpenAI
def example():
client = OpenAI(result="What are three activities to do in Paris?")
client.name = "Example"
client.user = "user123"
client.session = "session123"
completion = client.chat.completions.create(
name="Paris Activities",
model="gpt-4o",
temperature=0.7,
messages=[
{
"role": "user",
"content": "What are three activities to do in Paris?"
}
],
)
client.result = "Done"
Evals
You can perform evals directly on a completion object. The includes
eval is used here as an example, and checks if a string is included in the completion's output. The argument passed to eval()
is a name or label used for your reference.
from baserun import OpenAI
def example():
client = OpenAI()
completion = client.chat.completions.create(
model="gpt-4o",
temperature=0.7,
messages=[
{
"role": "user",
"content": "What are three activities to do in Paris?"
}
],
)
client.eval("include_eiffel_tower").includes("Eiffel Tower")
Tags
You can add tags either to the traced OpenAI object or to the completion. There are several different types of tags:
log
: Any arbitrary logs you want to attach to a trace or completionfeedback
: Any score-based feedback given from users (e.g. thumbs up/down, star rating)variable
: Any variables used, e.g. while rendering a templateannotate
: Any arbitrary attributes you want to attach to a trace or completion
Each tag type has functions on traced OpenAI objects and completions. Each tag function can accept a metadata
parameter which is an arbitrary dictionary with any values you might want to capture.
from baserun import OpenAI
def example():
client = OpenAI()
client.log("Gathering user input")
city = input()
completion = client.chat.completions.create(
model="gpt-4o",
temperature=0.7,
messages=[
{
"role": "user",
"content": f"What are three activities to do in {city}?"
}
],
)
completion.variable("city", city)
user_score = input()
client.feedback("User Score", score=user_score, metadata={"My key": "My value"})
Further Documentation
For a deeper dive on all capabilities and more advanced usage, please refer to our Documentation.
License
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for baserun-1.0.0b5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a588fa1ad09f0186064ff2f50ae40274c317f503bd09ea53d21c3302a274bc55 |
|
MD5 | 4951e5cec45b9f6578b9c6344c37a696 |
|
BLAKE2b-256 | 20140699bd97a23efda52370239fed84ea612e7029d74d89c2d422e051110001 |