mona-openai 0.2.1

Integration client for monitoring OpenAI usage with Mona

Mona-OpenAI Integration Client

Use one line of code to get instant live monitoring for your OpenAI usage including:

• Tokens usage
• Hallucination alerts
• Profanity and privacy analyses
• Behavioral drifts and anomalies
• LangChain support
• Much much more

Setting Up

$ pip install mona_openai

Quick Start

You can find boilerplate code for many use cases under the "examples" folder.

With Mona

Sign up for a free Mona account here.

from openai import OpenAI

from mona_openai import monitor_client

MONA_API_KEY = environ.get("MONA_API_KEY")
MONA_SECRET = environ.get("MONA_SECRET")
MONA_CREDS = {
    "key": MONA_API_KEY,
    "secret": MONA_SECRET,
}

# This is the name of the monitoring class on Mona
MONITORING_CONTEXT_NAME = "NEW_CHAT_CLIENT_CONTEXT"

openAI_client = monitor_client(OpenAI(api_key=environ.get("OPEN_AI_KEY")), MONA_CREDS, MONITORING_CONTEXT_NAME)

response = openAI_client.chat.completions.create(
  model="gpt-3.5-turbo",
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Who won the world series in 2020?"},
    {"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."},
    {"role": "user", "content": "Where was it played?"}
  ]
)
print(response.choices[0].message.content)

With Standard Logging

from openai import OpenAI
from os import environ

from mona_openai import monitor_client_with_logger

from mona_openai.loggers import StandardLogger
from logging import WARNING

MONA_API_KEY = environ.get("MONA_API_KEY")
MONA_SECRET = environ.get("MONA_SECRET")
MONA_CREDS = {
    "key": MONA_API_KEY,
    "secret": MONA_SECRET,
}

# This is the name of the monitoring class on Mona
MONITORING_CONTEXT_NAME = "NEW_CHAT_CLIENT_CONTEXT"

openAI_client = monitor_client_with_logger(OpenAI(api_key=environ.get("OPEN_AI_KEY")), StandardLogger(WARNING))

response = openAI_client.chat.completions.create(
  model="gpt-3.5-turbo",
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Who won the world series in 2020?"},
    {"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."},
    {"role": "user", "content": "Where was it played?"}
  ]
)
print(response.choices[0].message.content)

Supported OpenAI APIs

Currently this client supports just the Chat Completion API. Mona, not using this client, can support processes based on other APIs and also non-OpenAI-based apps. If you have a differrent use-case, we'd love to hear about it! Please email us at [e-mail].

Usage

Initialization

The main functions exposed in this package are monitor_client and monitor_client_with_logger.

These functions return an openai client that wraps the original chat completion method with an equivalent API that also logs relevant metrics for monitoring behind the scenes.

See above quick start examples for usage.

Specs

The specs arg allows you to configure what should be monitored. It expects a python dict with the follwoing possible keys:

• sampling_ratio (1): A number between 0 and 1 for how often should the call be logged.
• avoid_monitoring_exceptions (False): Whether or not to log out to Mona when there is an OpenAI exception. Default is to track exceptions - and Mona will alert you on things like a jump in number of exceptions
• export_prompt (False): Whether Mona should export the actual prompt text. Be default set to False to avoid privacy concerns.
• export_response_texts (False): Whether Mona should export the actual response texts. Be default set to False to avoid privacy concerns.
• analysis: A dictionary mapping each analysis type to a boolean value telling the client whether or not to run said analysis and log it to Mona. Possible options currently are "privacy", "profanity", and "textual". By default, all analyses take place and are logged out to Mona.

Using custom loggers

You don't have to have a Mona account to use this package. You can define specific loggers to log out the data to a file, memory, or just a given python logger.

This SDK provides a simple interface to implement your own loggers by inheriting from Logger under loggers/logger.py. Alternatively, by using the standard python logging library as in the example, you can create logging handlers to log the data out to any mechanism you choose (e.g., Kafka, Logstash, etc...)

Mona arguments you can add to the API call

• MONA_context_id: The unique id of the context in which the call is made. By using this ID you can export more data to Mona to the same context from other places. If not supplied, the "id" field of the OpenAI Endpoint's response will be used as the Mona context ID automatically.
• MONA_export_timestamp: Can be used to simulate as if the current call was made in a different time, as far as Mona is concerned.
• MONA_additional_data: A JSON-serializable dict with any other data you want to add to the monitoring context. This comes in handy if you want to add more information to the monitoring contex that isn't part of the basic OpenAI API call information. For example, if you are using a specific template ID or if this call is being made for a specific customer ID, these are fields you can add there to help get full context when monitoring with Mona.

Using OpenAI with REST calls instead of OpenAI's Python client

See rest examples in legacy example folder

Stream support

OpenAI allows receiving responses as a stream of tokens using the "stream" parameter. When this is done, Mona will collect all the tokens in memory (without interrupting the streaming process) and will create the analysis and log out the data the moment the stream is over. You don't need to do anything to make this happen.

Since for streaming responses OpenAI doesn't supply the full usage tokens summary, Mona uses the tiktoken package to calculate the tokens of the prompt and completion and log them for monitoring.

NOTE: Stream is currently only supported with SDK usage, and not with using REST directly.

Legacy LangChain support

You can use the exported monitor_langchain_llm to wrap a LangChain OpenAI LLM (chat or normal) with Mona's monitoring capabilities:

from mona_openai import monitor_langchain_llm

from langchain.llms import OpenAI

# Wrap the LLM object with Mona monitoring.
llm = monitor_langchain_llm(
    OpenAI(OPEN_AI_KEY),
    MONA_CREDS,
    MONITORING_CONTEXT_NAME)

See full example in completion_langchain.py in the examples folder.

Mona SDK

This package uses the mona_sdk package to export the relevant data to Mona. There are several environment variables you can use to configure the SDK's behavior. For example, you can set it up to raise exceptions when exporting data to Mona fails (it doesn't do that by default).

Monitoring for profanity

Mona uses the alt-profanity-check pacakge (https://pypi.org/project/alt-profanity-check/) to create both boolean predictions and probabilty scores for the existence of profanity both in the prompt and in the responses. We use the built in package methods for that. If you want, for example, to use a different probability threshold for the boolean prediction, you can do that by changing your Mona config on the Mona dashboard.

Using nest-asyncio

In environments in which there's a forever running event loop (e.g., Jupyter notebooks), the client might use nest_asyncio.apply() to run joint sync and async code.