Skip to main content

Wrapper library for openai to send events to Libretto

Project description

Libretto OpenAI Client

image

A drop-in replacement of openai.Client for sending events to Libretto.

Features

  • Provides a patched openai.Client that allows for setting Libretto-specific parameters for each request.
  • Currently supports the synchronous versions of completions.create() and chat.completions.create().

Get Started

To send events to Libretto, you'll need to create a project. From the project you'll need two things:

  1. API key: (api_key) This is generated for the project and is used to identify the project and environment (dev, staging, prod) that the event is coming from.
  2. Template Name: (prompt_template_name) This uniquely identifies a particular prompt that you are using and allows projects to have multiple prompts. This can be in any format but we recommend using a dash-separated format, e.g. my-prompt-name.

Note: Prompt template names can be auto-generated if the allow_unnamed_prompts configuration option is set (see below). However, if you rely on auto-generated names, new revisions of the same prompt will show up as different prompt templates in Libretto.

Usage

You can use the libretto_openai.Client anywhere that you're currently using the official openai.Client.

When instantiating a libretto_openai.Client, you can/should provide any of the existing openai.Client parameters in the constructor. Libretto-specific configuration can be provided via an additional libretto argument (see below).

To allow our tools to separate the "prompt" from the "prompt parameters", use TemplateChat and TemplateText to create templates.

Use TemplateChat For the ChatCompletion APIs:

from libretto_openai import (
    Client,
    LibrettoConfig,
    LibrettoCreateParams,
    TemplateChat,
)

client = Client(
    api_key="<OpenAI API Key>",
    libretto=LibrettoConfig(
        api_key="<Libretto API Key>",
    ),
)

completion = client.chat.completions.create(
    # Standard OpenAI parameters
    model="gpt-3.5-turbo",
    messages=TemplateChat(
        [{"role": "user", "content": "Show me an emoji that matches the sport: {sport}"}],
        {"sport": "soccer"},
    ),
    libretto=LibrettoCreateParams(
        prompt_template_name="sport-emoji",
    ),
)

Use TemplateText for the Completion API:

from libretto_openai import (
    Client,
    LibrettoConfig,
    LibrettoCreateParams,
    TemplateChat,
)

client = Client(
    api_key="<OpenAI API Key>",
    libretto=LibrettoConfig(
        api_key="<Libretto API Key>",
    ),
)

completion = client.completions.create(
    # Standard OpenAI parameters
    model="text-davinci-003",
    prompt=TemplateText(
        "Show me an emoji that matches the sport: {sport}",
        {"sport": "soccer"},
    ),
    libretto=LibrettoCreateParams(
        prompt_template_name="sport-emoji",
    ),
)

Advanced usage

Manually passing parameters

While the use of TemplateText and TemplateChat are preferred, you can optionally specify template data inline when calling the create() method:

completion = openai.ChatCompletion.create(
    model="gpt-3.5-turbo",

    # Note we are passing the raw messages object here
    messages=[{"role": "user", "content": "Show me an emoji that matches the sport: soccer"}],

    libretto=LibrettoCreateParams(
        # call configuration
        prompt_template_name="sport-emoji",

        # Here the prompt and parameters are passed seperately
        template_params={"sport": "soccer"},
        template_chat=[
            {"role": "user", "content": "Show me an emoji that matches the sport: {sport}"}
        ],
    ),
)

Configuration

The libretto kwarg that's present on the Client constructor is a LibrettoConfig object with the following options:

  • prompt_template_name: A default name to associate with prompts. If provided, this is the name that will be associated with any create call that's made without a libretto.prompt_template_name parameter.
  • allow_unnamed_prompts: When set to True, every prompt will be sent to Libretto even if no prompt template name as been provided (either via the prompt_template_name kwarg or via the libretto.prompt_template_name parameter on create). False by default.
  • redact_pii: When True, certain personally identifying information (PII) will be attempted to be redacted before being sent to the Libretto backend. See the pii package for details about the types of PII being detected/redacted. False by default.

Additional Create Call Parameters

When calling create(), a libretto argument should be provided to give Libretto-specific context to the call. The following parameters maybe specified:

  • chat_id: The id of a "chat session" - if the chat API is being used in a conversational context, then the same chat id can be provided so that the events are grouped together, in order. If not provided, this will be left blank.

  • template_chat: The chat template to record for chat requests. This is a list of dictionaries with the following keys:

    • role: The role of the speaker. Either "system", "user" or "ai".
    • content: The content of the message. This can be a string or a template string with {} placeholders.

    For example:

    completion = client.chat.completions.create(
        ...,
        libretto=LibrettoCreateParams(
            template_chat=[
                {"role": "ai", "content": "Hello, I'm {system_name}!"},
                {"role": "user", "content": "Hi {system_name}, I'm {user_name}!"}
            ],
        ),
    )
    

    To represent an array of chat messages, use the artificial role "chat_history" with content set to the variable name in substitution format: [{"role": "chat_history", "content": "{prev_messages}"}}]

  • template_text: The text template to record for completion requests. This is a string or a template string with {} placeholders.

    For example:

    completion = client.completions.create(
        ...,
        libretto=LibrettoCreateParams(
            template_text="Please welcome the user to {system_name}!",
        ),
    )
    
  • template_params: The parameters to use for template strings. This is a dictionary of key-value pairs.

    For example:

    completion = client.completions.create(
        ...,
        libretto=LibrettoCreateParams(
            template_text="Please welcome the user to {system_name}!",
            template_params={"system_name": "Awesome Comics Incorporated"},
        ),
    )
    
  • event_id: A unique UUID for a specific call. If not provided, one will be generated.

    For example:

    import uuid
    
    completion = client.completions.create(
        ...,
        libretto=LibrettoCreateParams(
            event_id=uuid.uuid4(),
        ),
    )
    
  • parent_event_id: The UUID of the parent event. All calls with the same parent id are grouped as a "Run Group".

    For example:

    import uuid
    
    parent_id = uuid.uuid4()
    # First call in the run group
    completion = client.completions.create(
        ...,
        libretto=LibrettoCreateParams(
            parent_event_id=parent_id,
        ),
    )
    
    # Another call in the same group
    completion = client.completions.create(
        ...,
        libretto=LibrettoCreateParams(
            parent_event_id=parent_id,
        ),
    )
    

Sending Feedback

Sometimes the answer provided by the LLM is not ideal, and your users may be able to help you find better responses. There are a few common cases:

  • You might use the LLM to suggest the title of a news article, but let the user edit it. If they change the title, you can send feedback to Libretto that the answer was not ideal.
  • You might provide a chatbot that answers questions, and the user can rate the answers with a thumbs up (good) or thumbs down (bad).

You can send this feedback to Libretto by calling send_feedback(). This will send a feedback event to Libretto about a prompt that was previously called, and let you review this feedback in the Libretto dashboard. You can use this feedback to develop new tests and improve your prompts.

from libretto_openai import Client

client = Client()
completion = client.completions.create(...)

# Maybe the user didn't like the answer, so ask them for a better one
better_response = ask_user_for_better_response(completion.choices[0].text)

# If the user provided a better answer, send feedback to Libretto
if better_response !== completion.choices[0].text:
    # feedback key is automatically injected into OpenAI response object as an extra field
    feedback_key = completion.model_extra.get("libretto_feedback_key")
    client.send_feedback(
        feedback_key=feedback_key,
        # Better answer from the user
        better_response=better_response,
        # Rating of existing answer, from 0 to 1
        rating=0.2,
    )

Note that feedback can include either rating, better_response, or both.

Parameters:

  • rating - a value from 0 (meaning the result was completely wrong) to 1 (meaning the result was correct)
  • better_response - the better response from the user

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

libretto_openai-1.0.4.tar.gz (59.0 kB view details)

Uploaded Source

Built Distribution

libretto_openai-1.0.4-py3-none-any.whl (61.0 kB view details)

Uploaded Python 3

File details

Details for the file libretto_openai-1.0.4.tar.gz.

File metadata

  • Download URL: libretto_openai-1.0.4.tar.gz
  • Upload date:
  • Size: 59.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.0.0 CPython/3.12.3

File hashes

Hashes for libretto_openai-1.0.4.tar.gz
Algorithm Hash digest
SHA256 a6cc952ce55666767df162b523eefe80c6ac15ebccf7bf5da1dd67d8f92339cb
MD5 0d2e271083d51d0b349ab561db05bfc8
BLAKE2b-256 92ce3dacdfa84d76871c65a8cc69425f556720b93155ade7330bb627cf2ba8a8

See more details on using hashes here.

File details

Details for the file libretto_openai-1.0.4-py3-none-any.whl.

File metadata

File hashes

Hashes for libretto_openai-1.0.4-py3-none-any.whl
Algorithm Hash digest
SHA256 e4b73d91c39d9f79455bf713e453b8d701f0324a616bb928c023b5cbe73bfa31
MD5 be87fd8d472ca61caee90a1b361a0978
BLAKE2b-256 557cc3ec62c6a46b4805379636f50c49ffb0bc6d23e1ab6ef212c0992a46a85c

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page