promptwatch.io python client to trace langchain sessions
Project description
PromptWatch.io ... session tracking for LangChain
It enables you to:
- track all the chain executions
- track LLM Prompts and re-play the LLM runs with the same input parameters and model settings to tweak your prompt template
- track your costs per project and per tenant (your customer)
Installation
pip install promptwatch
Basic usage
In order to enable session tracking wrap you chain executions in PromptWatch block
from langchain import OpenAI, LLMChain, PromptTemplate
from promptwatch import PromptWatch
prompt_template = PromptTemplate.from_template("Finish this sentence {input}")
my_chain = LLMChain(llm=OpenAI(), prompt=prompt_template)
with PromptWatch(api_key="<your-api-key>") as pw:
my_chain("The quick brown fox jumped over")
Here you can get your API key: http://www.promptwatch.io/get-api-key (no registration needed)
You can set it directly into PromptWatch
constructor, or set is as an ENV variable PROMPTWATCH_API_KEY
Project and Tenant costs tracking
You can assign a project and tenant id to your session by setting the constructor parameter:
This will allow you to track costs per OpenAI request per customer and as well as your dev project.
...
with PromptWatch(tracking_project="my-project", tracking_tenant="my-tenant",) as pw:
my_chain("The quick brown fox jumped over")
What is being tracked
PromptWatch tracks all the details that LangChain exposes via its tracking "API" and more.
👉 Chain execution inputs, outputs, execution time
👉 Tools input output
👉 retrieved documents from retrieval vector DB
👉 Details about LLM runs like:
- final prompt text
- generated text
- execution details like model, temperature, etc. (everything you need to re-run the prompt with the same exact setup)
- total used tokens
- costs (based on OpenAI price list per model)
- prompt template and its parameters
Custom logging
PromptWatch tracks quite extensively standard LangChain tools, but if you have some custom code you'd like to track you can do so.
...
with PromptWatch(api_key=invalid_api_key):
PromptWatch.log_activity(Question(text="What did the president say about Ketanji Brown Jackson"))
PromptWatch.log("my arbitrary log message")
All the logs are associated with to opened session. You can't log outside the session.
...
with PromptWatch(api_key=invalid_api_key):
PromptWatch.log_activity(Question(text="What did the president say about Ketanji Brown Jackson"))
#end of session
PromptWatch.log("this will raise an exception!")
Prompt template tracking
You can register any LangChain prompt template for detailed monitoring
from promptwatch import PromptWatch, register_prompt_template
from langchain import OpenAI, LLMChain, PromptTemplate
prompt_template = PromptTemplate.from_template("Finish this sentence {input}")
my_chain = LLMChain(llm=OpenAI(), prompt=prompt_template)
register_prompt_template("your_template_name",prompt_template)
with PromptWatch() as pw:
#execute the chain
my_chain("The quick brown fox jumped over")
This will allow you to associate the prompt template with a custom name (and function) and track it independently...
Any change of that template text will cause an automatic version change (with automatic version number increment)
Warning The registration just assigns the template a custom name and is only done when the LLM actually executes the LLM prompt with that template. Therefore is has no additional performance costs, on the contrary it can even speed up the execution a bit if the same template is used multiple times.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file promptwatch-0.0.5.tar.gz
.
File metadata
- Download URL: promptwatch-0.0.5.tar.gz
- Upload date:
- Size: 20.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.9.13
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | ac5ac440371c0664171f4f0deeba11863dc89ed327754767ddc1ffff71a2e9bf |
|
MD5 | dde5f6eab26d49de58bbf64232668619 |
|
BLAKE2b-256 | 70ee250389a506ff991d4d7edd5214a0d0b941f08b53c8444c02fac51eab2a0e |
File details
Details for the file promptwatch-0.0.5-py3-none-any.whl
.
File metadata
- Download URL: promptwatch-0.0.5-py3-none-any.whl
- Upload date:
- Size: 28.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.9.13
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 73657f42606d28e8aa4b390d0ad31769b99220452c6ee75b3e0c16d8f9f5115a |
|
MD5 | 0e42253eb78a62c7ba963c8b72a59e71 |
|
BLAKE2b-256 | 12b007562ea9baff984e4ee601c7b0d74f6b59c6f2110b85f1b3487b2bebd193 |