Skip to main content

Wrapper library for openai to send events to Libretto

Project description

Libretto OpenAI Client

image

A drop-in replacement of openai.Client for sending events to Libretto.

Features

  • Provides a patched openai.Client that allows for setting Libretto-specific parameters for each request.
  • Currently supports the synchronous versions of completions.create() and chat.completions.create().

Get Started

To send events to Libretto, you'll need to create a project. From the project you'll need two things:

  1. API key: (api_key) This is generated for the project and is used to identify the project and environment (dev, staging, prod) that the event is coming from.
  2. Template Name: (prompt_template_name) This uniquely identifies a particular prompt that you are using and allows projects to have multiple prompts. This can be in any format but we recommend using a dash-separated format, e.g. my-prompt-name.

Note: Prompt template names can be auto-generated if the allow_unnamed_prompts configuration option is set (see below). However, if you rely on auto-generated names, new revisions of the same prompt will show up as different prompt templates in Libretto.

Usage

You can use the libretto_openai.Client anywhere that you're currently using the official openai.Client.

When instantiating a libretto_openai.Client, you can/should provide any of the existing openai.Client parameters in the constructor. Libretto-specific configuration can be provided via an additional libretto argument (see below).

To allow our tools to separate the "prompt" from the "prompt parameters", use TemplateChat and TemplateText to create templates.

Use TemplateChat For the ChatCompletion APIs:

from libretto_openai import (
    Client,
    LibrettoConfig,
    LibrettoCreateParams,
    TemplateChat,
)

client = Client(
    api_key="<OpenAI API Key>",
    libretto=LibrettoConfig(
        api_key="<Libretto API Key>",
    ),
)

completion = client.chat.completions.create(
    # Standard OpenAI parameters
    model="gpt-4o-mini",
    messages=TemplateChat(
        [{"role": "user", "content": "Show me an emoji that matches the sport: {sport}"}],
        {"sport": "soccer"},
    ),
    libretto=LibrettoCreateParams(
        prompt_template_name="sport-emoji",
    ),
)

Use TemplateText for the Completion API:

from libretto_openai import (
    Client,
    LibrettoConfig,
    LibrettoCreateParams,
    TemplateChat,
)

client = Client(
    api_key="<OpenAI API Key>",
    libretto=LibrettoConfig(
        api_key="<Libretto API Key>",
    ),
)

completion = client.completions.create(
    # Standard OpenAI parameters
    model="text-davinci-003",
    prompt=TemplateText(
        "Show me an emoji that matches the sport: {sport}",
        {"sport": "soccer"},
    ),
    libretto=LibrettoCreateParams(
        prompt_template_name="sport-emoji",
    ),
)

Advanced usage

Manually passing parameters

While the use of TemplateText and TemplateChat are preferred, you can optionally specify template data inline when calling the create() method:

completion = openai.ChatCompletion.create(
    model="gpt-4o-mini",

    # Note we are passing the raw messages object here
    messages=[{"role": "user", "content": "Show me an emoji that matches the sport: soccer"}],

    libretto=LibrettoCreateParams(
        # call configuration
        prompt_template_name="sport-emoji",

        # Here the prompt and parameters are passed seperately
        template_params={"sport": "soccer"},
        template_chat=[
            {"role": "user", "content": "Show me an emoji that matches the sport: {sport}"}
        ],
    ),
)

Configuration

The libretto kwarg that's present on the Client constructor is a LibrettoConfig object with the following options:

  • prompt_template_name: A default name to associate with prompts. If provided, this is the name that will be associated with any create call that's made without a libretto.prompt_template_name parameter.
  • allow_unnamed_prompts: When set to True, every prompt will be sent to Libretto even if no prompt template name as been provided (either via the prompt_template_name kwarg or via the libretto.prompt_template_name parameter on create). False by default.
  • redact_pii: When True, certain personally identifying information (PII) will be attempted to be redacted before being sent to the Libretto backend. See the pii package for details about the types of PII being detected/redacted. False by default.

Additional Create Call Parameters

When calling create(), a libretto argument should be provided to give Libretto-specific context to the call. The following parameters maybe specified:

  • chat_id: The id of a "chat session" - if the chat API is being used in a conversational context, then the same chat id can be provided so that the events are grouped together, in order. If not provided, this will be left blank.

  • template_chat: The chat template to record for chat requests. This is a list of dictionaries with the following keys:

    • role: The role of the speaker. Either "system", "user" or "ai".
    • content: The content of the message. This can be a string or a template string with {} placeholders.

    For example:

    completion = client.chat.completions.create(
        ...,
        libretto=LibrettoCreateParams(
            template_chat=[
                {"role": "ai", "content": "Hello, I'm {system_name}!"},
                {"role": "user", "content": "Hi {system_name}, I'm {user_name}!"}
            ],
        ),
    )
    

    To represent an array of chat messages, use the artificial role "chat_history" with content set to the variable name in substitution format: [{"role": "chat_history", "content": "{prev_messages}"}}]

  • template_text: The text template to record for completion requests. This is a string or a template string with {} placeholders.

    For example:

    completion = client.completions.create(
        ...,
        libretto=LibrettoCreateParams(
            template_text="Please welcome the user to {system_name}!",
        ),
    )
    
  • template_params: The parameters to use for template strings. This is a dictionary of key-value pairs.

    For example:

    completion = client.completions.create(
        ...,
        libretto=LibrettoCreateParams(
            template_text="Please welcome the user to {system_name}!",
            template_params={"system_name": "Awesome Comics Incorporated"},
        ),
    )
    
  • event_id: A unique UUID for a specific call. If not provided, one will be generated.

    For example:

    import uuid
    
    completion = client.completions.create(
        ...,
        libretto=LibrettoCreateParams(
            event_id=uuid.uuid4(),
        ),
    )
    
  • chain_id: A UUID that groups related events together in a chain. For example, if you have a multi-step workflow where one LLM call's output feeds into another LLM call, you can use the same chainId to track the full sequence of events.

    For example:

    import uuid
    
    parent_id = uuid.uuid4()
    
    # First call in the run group
    completion = client.completions.create(
        ...,
        libretto=LibrettoCreateParams(
            chain_id=parent_id,
        ),
    )
    
    # Another call in the same group
    completion = client.completions.create(
        ...,
        libretto=LibrettoCreateParams(
            chain_id=parent_id,
        ),
    )
    

Sending Feedback

Sometimes the answer provided by the LLM is not ideal, and your users may be able to help you find better responses. There are a few common cases:

  • You might use the LLM to suggest the title of a news article, but let the user edit it. If they change the title, you can send feedback to Libretto that the answer was not ideal.
  • You might provide a chatbot that answers questions, and the user can rate the answers with a thumbs up (good) or thumbs down (bad).

You can send this feedback to Libretto by calling send_feedback(). This will send a feedback event to Libretto about a prompt that was previously called, and let you review this feedback in the Libretto dashboard. You can use this feedback to develop new tests and improve your prompts.

from libretto_openai import Client

client = Client()
completion = client.completions.create(...)

# Maybe the user didn't like the answer, so ask them for a better one
better_response = ask_user_for_better_response(completion.choices[0].text)

# If the user provided a better answer, send feedback to Libretto
if better_response !== completion.choices[0].text:
    # feedback key is automatically injected into OpenAI response object as an extra field
    feedback_key = completion.model_extra.get("libretto_feedback_key")
    client.send_feedback(
        feedback_key=feedback_key,
        # Better answer from the user
        better_response=better_response,
        # Rating of existing answer, from 0 to 1
        rating=0.2,
    )

Note that feedback can include either rating, better_response, or both.

Parameters:

  • rating - a value from 0 (meaning the result was completely wrong) to 1 (meaning the result was correct)
  • better_response - the better response from the user

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

libretto_openai-1.1.3.tar.gz (59.1 kB view details)

Uploaded Source

Built Distribution

libretto_openai-1.1.3-py3-none-any.whl (61.5 kB view details)

Uploaded Python 3

File details

Details for the file libretto_openai-1.1.3.tar.gz.

File metadata

  • Download URL: libretto_openai-1.1.3.tar.gz
  • Upload date:
  • Size: 59.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for libretto_openai-1.1.3.tar.gz
Algorithm Hash digest
SHA256 408bc06932152d7085407e4ff2e1a4ad52c7ee757074a2f3e8da53d8ff34be86
MD5 7c1b522ea60c7942bb197b1c9670e844
BLAKE2b-256 c7e4faf011b44455abaa0e4d81a232e2cb235a682a250069e86191bfe5341dce

See more details on using hashes here.

Provenance

The following attestation bundles were made for libretto_openai-1.1.3.tar.gz:

Publisher: release.yml on libretto-ai/openai-py

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libretto_openai-1.1.3-py3-none-any.whl.

File metadata

File hashes

Hashes for libretto_openai-1.1.3-py3-none-any.whl
Algorithm Hash digest
SHA256 bfdb51ea0fc152f834e4a683f16a677ee708ce2d325a9bde8e48ed543c0f5f8d
MD5 2e208285884c72115bf05a86d94b981b
BLAKE2b-256 73f420266c9075fba84a6120778a9f998cea2c48a17e92863e027acdc7cd2d7b

See more details on using hashes here.

Provenance

The following attestation bundles were made for libretto_openai-1.1.3-py3-none-any.whl:

Publisher: release.yml on libretto-ai/openai-py

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page