Skip to main content

promptwatch.io python client to trace langchain sessions

Project description

PromptWatch.io ... session tracking for LangChain

It enables you to:

  • track all the chain executions
  • track LLM Prompts and re-play the LLM runs with the same input parameters and model settings to tweak your prompt template
  • track your costs per project and per tenant (your customer)

Installation

pip install promptwatch

Basic usage

In order to enable session tracking wrap you chain executions in PromptWatch block

from langchain import OpenAI, LLMChain, PromptTemplate
from promptwatch import PromptWatch

prompt_template = PromptTemplate.from_template("Finish this sentence {input}")
my_chain = LLMChain(llm=OpenAI(), prompt=prompt_template)

with PromptWatch(api_key="<your-api-key>") as pw:
    my_chain("The quick brown fox jumped over")

Here you can get your API key: http://www.promptwatch.io/get-api-key (no registration needed)

You can set it directly into PromptWatch constructor, or set is as an ENV variable PROMPTWATCH_API_KEY

Project and Tenant costs tracking

You can assign a project and tenant id to your session by setting the constructor parameter:

This will allow you to track costs per OpenAI request per customer and as well as your dev project.

...

with PromptWatch(tracking_project="my-project", tracking_tenant="my-tenant",) as pw:
    my_chain("The quick brown fox jumped over")

What is being tracked

PromptWatch tracks all the details that LangChain exposes via its tracking "API" and more.

👉 Chain execution inputs, outputs, execution time

👉 Tools input output

👉 retrieved documents from retrieval vector DB

👉 Details about LLM runs like:

  • final prompt text
  • generated text
  • execution details like model, temperature, etc. (everything you need to re-run the prompt with the same exact setup)
  • total used tokens
  • costs (based on OpenAI price list per model)
  • prompt template and its parameters

Custom logging

PromptWatch tracks quite extensively standard LangChain tools, but if you have some custom code you'd like to track you can do so.

...
with PromptWatch(api_key=invalid_api_key):
    PromptWatch.log_activity(Question(text="What did the president say about Ketanji Brown Jackson"))
    PromptWatch.log("my arbitrary log message")

All the logs are associated with to opened session. You can't log outside the session.

...
with PromptWatch(api_key=invalid_api_key):
    PromptWatch.log_activity(Question(text="What did the president say about Ketanji Brown Jackson"))
    #end of session   
PromptWatch.log("this will raise an exception!")

Prompt template tracking

You can register any LangChain prompt template for detailed monitoring

from promptwatch import PromptWatch, register_prompt_template
from langchain import OpenAI, LLMChain, PromptTemplate

prompt_template = PromptTemplate.from_template("Finish this sentence {input}")
my_chain = LLMChain(llm=OpenAI(), prompt=prompt_template)

register_prompt_template("your_template_name",prompt_template) 

with PromptWatch() as pw:
    
    #execute the chain
    my_chain("The quick brown fox jumped over")

This will allow you to associate the prompt template with a custom name (and function) and track it independently...

Any change of that template text will cause an automatic version change (with automatic version number increment)

Warning The registration just assigns the template a custom name and is only done when the LLM actually executes the LLM prompt with that template. Therefore is has no additional performance costs, on the contrary it can even speed up the execution a bit if the same template is used multiple times.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

promptwatch-0.0.8.tar.gz (20.6 kB view details)

Uploaded Source

Built Distribution

promptwatch-0.0.8-py3-none-any.whl (28.9 kB view details)

Uploaded Python 3

File details

Details for the file promptwatch-0.0.8.tar.gz.

File metadata

  • Download URL: promptwatch-0.0.8.tar.gz
  • Upload date:
  • Size: 20.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.9.16

File hashes

Hashes for promptwatch-0.0.8.tar.gz
Algorithm Hash digest
SHA256 dd9f7388bb9aa23aecb8b75ef260589960cf5febcd9b6f93fc5ff8a808993b7b
MD5 9f7b1966a2ba244af303bb03e17e2608
BLAKE2b-256 19452efd0a483d0536ff10df3768c914480d2dd59f7aa05ccb08dc07bf69ed2a

See more details on using hashes here.

File details

Details for the file promptwatch-0.0.8-py3-none-any.whl.

File metadata

  • Download URL: promptwatch-0.0.8-py3-none-any.whl
  • Upload date:
  • Size: 28.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.9.16

File hashes

Hashes for promptwatch-0.0.8-py3-none-any.whl
Algorithm Hash digest
SHA256 161dac55a7f08660df4809f7b7f98cc71fe9ec9abebf998e364369533976c1a8
MD5 cd18b407959e8f478090be1b07619891
BLAKE2b-256 c29756621bf9ccb2800a3a63f5ecca2ba92de42b1d9c790c6c2b5f08b630dd86

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page