Skip to main content

promptwatch.io python client to trace langchain sessions

Project description

PromptWatch.io ... session tracking for LangChain

It enables you to:

  • track all the chain executions
  • track LLM Prompts and re-play the LLM runs with the same input parameters and model settings to tweak your prompt template
  • track your costs per project and per tenant (your customer)

Installation

pip install promptwatch

Basic usage

In order to enable session tracking wrap you chain executions in PromptWatch block

from langchain import OpenAI, LLMChain, PromptTemplate
from promptwatch import PromptWatch

prompt_template = PromptTemplate.from_template("Finish this sentence {input}")
my_chain = LLMChain(llm=OpenAI(), prompt=prompt_template)

with PromptWatch(api_key="<your-api-key>") as pw:
    my_chain("The quick brown fox jumped over")

Here you can get your API key: http://www.promptwatch.io/get-api-key (no registration needed)

You can set it directly into PromptWatch constructor, or set is as an ENV variable PROMPTWATCH_API_KEY

Project and Tenant costs tracking

You can assign a project and tenant id to your session by setting the constructor parameter:

This will allow you to track costs per OpenAI request per customer and as well as your dev project.

...

with PromptWatch(tracking_project="my-project", tracking_tenant="my-tenant",) as pw:
    my_chain("The quick brown fox jumped over")

What is being tracked

PromptWatch tracks all the details that LangChain exposes via its tracking "API" and more.

👉 Chain execution inputs, outputs, execution time

👉 Tools input output

👉 retrieved documents from retrieval vector DB

👉 Details about LLM runs like:

  • final prompt text
  • generated text
  • execution details like model, temperature, etc. (everything you need to re-run the prompt with the same exact setup)
  • total used tokens
  • costs (based on OpenAI price list per model)
  • prompt template and its parameters

Custom logging

PromptWatch tracks quite extensively standard LangChain tools, but if you have some custom code you'd like to track you can do so.

...
with PromptWatch(api_key=invalid_api_key):
    PromptWatch.log_activity(Question(text="What did the president say about Ketanji Brown Jackson"))
    PromptWatch.log("my arbitrary log message")

All the logs are associated with to opened session. You can't log outside the session.

...
with PromptWatch(api_key=invalid_api_key):
    PromptWatch.log_activity(Question(text="What did the president say about Ketanji Brown Jackson"))
    #end of session   
PromptWatch.log("this will raise an exception!")

Prompt template tracking

You can register any LangChain prompt template for detailed monitoring

from promptwatch import PromptWatch, register_prompt_template
from langchain import OpenAI, LLMChain, PromptTemplate

prompt_template = PromptTemplate.from_template("Finish this sentence {input}")
my_chain = LLMChain(llm=OpenAI(), prompt=prompt_template)

register_prompt_template("your_template_name",prompt_template) 

with PromptWatch() as pw:
    
    #execute the chain
    my_chain("The quick brown fox jumped over")

This will allow you to associate the prompt template with a custom name (and function) and track it independently...

Any change of that template text will cause an automatic version change (with automatic version number increment)

Warning The registration just assigns the template a custom name and is only done when the LLM actually executes the LLM prompt with that template. Therefore is has no additional performance costs, on the contrary it can even speed up the execution a bit if the same template is used multiple times.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

promptwatch-0.0.5.tar.gz (20.3 kB view details)

Uploaded Source

Built Distribution

promptwatch-0.0.5-py3-none-any.whl (28.7 kB view details)

Uploaded Python 3

File details

Details for the file promptwatch-0.0.5.tar.gz.

File metadata

  • Download URL: promptwatch-0.0.5.tar.gz
  • Upload date:
  • Size: 20.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.9.13

File hashes

Hashes for promptwatch-0.0.5.tar.gz
Algorithm Hash digest
SHA256 ac5ac440371c0664171f4f0deeba11863dc89ed327754767ddc1ffff71a2e9bf
MD5 dde5f6eab26d49de58bbf64232668619
BLAKE2b-256 70ee250389a506ff991d4d7edd5214a0d0b941f08b53c8444c02fac51eab2a0e

See more details on using hashes here.

File details

Details for the file promptwatch-0.0.5-py3-none-any.whl.

File metadata

  • Download URL: promptwatch-0.0.5-py3-none-any.whl
  • Upload date:
  • Size: 28.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.9.13

File hashes

Hashes for promptwatch-0.0.5-py3-none-any.whl
Algorithm Hash digest
SHA256 73657f42606d28e8aa4b390d0ad31769b99220452c6ee75b3e0c16d8f9f5115a
MD5 0e42253eb78a62c7ba963c8b72a59e71
BLAKE2b-256 12b007562ea9baff984e4ee601c7b0d74f6b59c6f2110b85f1b3487b2bebd193

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page