Skip to main content

Client for Humanloop API

Project description

Visit Humanloop

Humanloop

PyPI README.md

[!WARNING] This SDK has breaking changes in >= 0.6.0 versions. All methods now return Pydantic models.

Before (< 0.6.0)

Previously, you had to use the [] syntax to access response values. This required a little more code for every property access.

chat_response = humanloop.chat(
        # parameters
    )
print(chat_response.body["project_id"])

After (>= 0.6.0)

With Pydantic-based response values, you can use the . syntax to access. This is slightly less verbose and looks more Pythonic.

chat_response = humanloop.chat(
        # parameters
    )
print(chat_response.project_id)

To reuse existing implementations from < 0.6.0, use the .raw namespace as specified in the Raw HTTP Response section.

Table of Contents

Requirements

Python >=3.7

Installation

pip install humanloop==0.7.30

Getting Started

from pprint import pprint
from humanloop import Humanloop, ApiException

humanloop = Humanloop(
    api_key="YOUR_API_KEY",
    openai_api_key="YOUR_OPENAI_API_KEY",
    anthropic_api_key="YOUR_ANTHROPIC_API_KEY",
)

try:
    # Chat
    chat_response = humanloop.chat(
        project="sdk-example",
        messages=[
            {
                "role": "user",
                "content": "Explain asynchronous programming.",
            }
        ],
        model_config={
            "model": "gpt-3.5-turbo",
            "max_tokens": -1,
            "temperature": 0.7,
            "chat_template": [
                {
                    "role": "system",
                    "content": "You are a helpful assistant who replies in the style of {{persona}}.",
                },
            ],
        },
        inputs={
            "persona": "the pirate Blackbeard",
        },
        stream=False,
    )
    print(chat_response)
except ApiException as e:
    print("Exception when calling .chat: %s\n" % e)
    pprint(e.body)
    if e.status == 422:
        pprint(e.body["detail"])
    pprint(e.headers)
    pprint(e.status)
    pprint(e.reason)
    pprint(e.round_trip_time)

try:
    # Complete
    complete_response = humanloop.complete(
        project="sdk-example",
        inputs={
            "text": "Llamas that are well-socialized and trained to halter and lead after weaning and are very friendly and pleasant to be around. They are extremely curious and most will approach people easily. However, llamas that are bottle-fed or over-socialized and over-handled as youth will become extremely difficult to handle when mature, when they will begin to treat humans as they treat each other, which is characterized by bouts of spitting, kicking and neck wrestling.[33]",
        },
        model_config={
            "model": "gpt-3.5-turbo",
            "max_tokens": -1,
            "temperature": 0.7,
            "prompt_template": "Summarize this for a second-grade student:\n\nText:\n{{text}}\n\nSummary:\n",
        },
        stream=False,
    )
    print(complete_response)
except ApiException as e:
    print("Exception when calling .complete: %s\n" % e)
    pprint(e.body)
    if e.status == 422:
        pprint(e.body["detail"])
    pprint(e.headers)
    pprint(e.status)
    pprint(e.reason)
    pprint(e.round_trip_time)

try:
    # Feedback
    feedback_response = humanloop.feedback(
        type="rating",
        value="good",
        data_id="data_[...]",
        user="user@example.com",
    )
    print(feedback_response)
except ApiException as e:
    print("Exception when calling .feedback: %s\n" % e)
    pprint(e.body)
    if e.status == 422:
        pprint(e.body["detail"])
    pprint(e.headers)
    pprint(e.status)
    pprint(e.reason)
    pprint(e.round_trip_time)

try:
    # Log
    log_response = humanloop.log(
        project="sdk-example",
        inputs={
            "text": "Llamas that are well-socialized and trained to halter and lead after weaning and are very friendly and pleasant to be around. They are extremely curious and most will approach people easily. However, llamas that are bottle-fed or over-socialized and over-handled as youth will become extremely difficult to handle when mature, when they will begin to treat humans as they treat each other, which is characterized by bouts of spitting, kicking and neck wrestling.[33]",
        },
        output="Llamas can be friendly and curious if they are trained to be around people, but if they are treated too much like pets when they are young, they can become difficult to handle when they grow up. This means they might spit, kick, and wrestle with their necks.",
        source="sdk",
        config={
            "model": "gpt-3.5-turbo",
            "max_tokens": -1,
            "temperature": 0.7,
            "prompt_template": "Summarize this for a second-grade student:\n\nText:\n{{text}}\n\nSummary:\n",
            "type": "model",
        },
    )
    print(log_response)
except ApiException as e:
    print("Exception when calling .log: %s\n" % e)
    pprint(e.body)
    if e.status == 422:
        pprint(e.body["detail"])
    pprint(e.headers)
    pprint(e.status)
    pprint(e.reason)
    pprint(e.round_trip_time)

Async

async support is available by prepending a to any method.

import asyncio
from pprint import pprint
from humanloop import Humanloop, ApiException

humanloop = Humanloop(
    api_key="YOUR_API_KEY",
    openai_api_key="YOUR_OPENAI_API_KEY",
    anthropic_api_key="YOUR_ANTHROPIC_API_KEY",
)


async def main():
    try:
        complete_response = await humanloop.acomplete(
            project="sdk-example",
            inputs={
                "text": "Llamas that are well-socialized and trained to halter and lead after weaning and are very friendly and pleasant to be around. They are extremely curious and most will approach people easily. However, llamas that are bottle-fed or over-socialized and over-handled as youth will become extremely difficult to handle when mature, when they will begin to treat humans as they treat each other, which is characterized by bouts of spitting, kicking and neck wrestling.[33]",
            },
            model_config={
                "model": "gpt-3.5-turbo",
                "max_tokens": -1,
                "temperature": 0.7,
                "prompt_template": "Summarize this for a second-grade student:\n\nText:\n{{text}}\n\nSummary:\n",
            },
            stream=False,
        )
        print(complete_response)
    except ApiException as e:
        print("Exception when calling .complete: %s\n" % e)
        pprint(e.body)
        if e.status == 422:
            pprint(e.body["detail"])
        pprint(e.headers)
        pprint(e.status)
        pprint(e.reason)
        pprint(e.round_trip_time)


asyncio.run(main())

Raw HTTP Response

To access raw HTTP response values, use the .raw namespace.

from pprint import pprint
from humanloop import Humanloop, ApiException

humanloop = Humanloop(
    openai_api_key="OPENAI_API_KEY",
    openai_azure_api_key="OPENAI_AZURE_API_KEY",
    openai_azure_endpoint_api_key="OPENAI_AZURE_ENDPOINT_API_KEY",
    anthropic_api_key="ANTHROPIC_API_KEY",
    cohere_api_key="COHERE_API_KEY",
    api_key="YOUR_API_KEY",
)

try:
    # Chat
    create_response = humanloop.chats.raw.create(
        messages=[
            {
                "role": "user",
            }
        ],
        model_config={
            "provider": "openai",
            "model": "model_example",
            "max_tokens": -1,
            "temperature": 1,
            "top_p": 1,
            "presence_penalty": 0,
            "frequency_penalty": 0,
            "endpoint": "complete",
        },
        project="string_example",
        project_id="string_example",
        session_id="string_example",
        session_reference_id="string_example",
        parent_id="string_example",
        parent_reference_id="string_example",
        inputs={},
        source="string_example",
        metadata={},
        save=True,
        source_datapoint_id="string_example",
        provider_api_keys={},
        num_samples=1,
        stream=False,
        user="string_example",
        seed=1,
        return_inputs=True,
        tool_choice="string_example",
        tool_call="string_example",
        response_format={
            "type": "json_object",
        },
    )
    pprint(create_response.body)
    pprint(create_response.body["data"])
    pprint(create_response.body["provider_responses"])
    pprint(create_response.body["project_id"])
    pprint(create_response.body["num_samples"])
    pprint(create_response.body["logprobs"])
    pprint(create_response.body["suffix"])
    pprint(create_response.body["user"])
    pprint(create_response.body["usage"])
    pprint(create_response.body["metadata"])
    pprint(create_response.body["provider_request"])
    pprint(create_response.body["session_id"])
    pprint(create_response.body["tool_choice"])
    pprint(create_response.headers)
    pprint(create_response.status)
    pprint(create_response.round_trip_time)
except ApiException as e:
    print("Exception when calling ChatsApi.create: %s\n" % e)
    pprint(e.body)
    if e.status == 422:
        pprint(e.body["detail"])
    pprint(e.headers)
    pprint(e.status)
    pprint(e.reason)
    pprint(e.round_trip_time)

Streaming

Streaming support is available by suffixing a chat or complete method with _stream.

import asyncio
from humanloop import Humanloop

humanloop = Humanloop(
    api_key="YOUR_API_KEY",
    openai_api_key="YOUR_OPENAI_API_KEY",
    anthropic_api_key="YOUR_ANTHROPIC_API_KEY",
)


async def main():
    response = await humanloop.chat_stream(
        project="sdk-example",
        messages=[
            {
                "role": "user",
                "content": "Explain asynchronous programming.",
            }
        ],
        model_config={
            "model": "gpt-3.5-turbo",
            "max_tokens": -1,
            "temperature": 0.7,
            "chat_template": [
                {
                    "role": "system",
                    "content": "You are a helpful assistant who replies in the style of {{persona}}.",
                },
            ],
        },
        inputs={
            "persona": "the pirate Blackbeard",
        },
    )
    async for token in response.content:
        print(token)


asyncio.run(main())

Reference

humanloop.chat

Get a chat response by providing details of the model configuration in the request.

๐Ÿ› ๏ธ Usage

create_response = humanloop.chat(
    messages=[
        {
            "role": "user",
        }
    ],
    model_config={
        "provider": "openai",
        "model": "model_example",
        "max_tokens": -1,
        "temperature": 1,
        "top_p": 1,
        "presence_penalty": 0,
        "frequency_penalty": 0,
        "endpoint": "complete",
    },
    project="string_example",
    project_id="string_example",
    session_id="string_example",
    session_reference_id="string_example",
    parent_id="string_example",
    parent_reference_id="string_example",
    inputs={},
    source="string_example",
    metadata={},
    save=True,
    source_datapoint_id="string_example",
    provider_api_keys={},
    num_samples=1,
    stream=False,
    user="string_example",
    seed=1,
    return_inputs=True,
    tool_choice="string_example",
    tool_call="string_example",
    response_format={
        "type": "json_object",
    },
)

โš™๏ธ Parameters

messages: List[ChatMessageWithToolCall]

The messages passed to the to provider chat endpoint.

model_config: ModelConfigChatRequest

The model configuration used to create a chat response.

project: str

Unique project name. If no project exists with this name, a new project will be created.

project_id: str

Unique ID of a project to associate to the log. Either this or project must be provided.

session_id: str

ID of the session to associate the datapoint.

session_reference_id: str

A unique string identifying the session to associate the datapoint to. Allows you to log multiple datapoints to a session (using an ID kept by your internal systems) by passing the same session_reference_id in subsequent log requests. Specify at most one of this or session_id.

parent_id: str

ID associated to the parent datapoint in a session.

parent_reference_id: str

A unique string identifying the previously-logged parent datapoint in a session. Allows you to log nested datapoints with your internal system IDs by passing the same reference ID as parent_id in a prior log request. Specify at most one of this or parent_id. Note that this cannot refer to a datapoint being logged in the same request.

inputs: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

The inputs passed to the prompt template.

source: str

Identifies where the model was called from.

metadata: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

Any additional metadata to record.

save: bool

Whether the request/response payloads will be stored on Humanloop.

source_datapoint_id: str

ID of the source datapoint if this is a log derived from a datapoint in a dataset.

provider_api_keys: ProviderApiKeys

API keys required by each provider to make API calls. The API keys provided here are not stored by Humanloop. If not specified here, Humanloop will fall back to the key saved to your organization.

num_samples: int

The number of generations.

stream: bool

If true, tokens will be sent as data-only server-sent events. If num_samples > 1, samples are streamed back independently.

user: str

End-user ID passed through to provider call.

seed: int

Deprecated field: the seed is instead set as part of the request.config object.

return_inputs: bool

Whether to return the inputs in the response. If false, the response will contain an empty dictionary under inputs. This is useful for reducing the size of the response. Defaults to true.

tool_choice: Union[str, str, str, ToolChoice]

Controls how the model uses tools. The following options are supported: 'none' forces the model to not call a tool; the default when no tools are provided as part of the model config. 'auto' the model can decide to call one of the provided tools; the default when tools are provided as part of the model config. Providing {'type': 'function', 'function': {name': <TOOL_NAME>}} forces the model to use the named function.

tool_call: Union[str, Dict[str, str]]

NB: Deprecated with new tool_choice. Controls how the model uses tools. The following options are supported: 'none' forces the model to not call a tool; the default when no tools are provided as part of the model config. 'auto' the model can decide to call one of the provided tools; the default when tools are provided as part of the model config. Providing {'name': <TOOL_NAME>} forces the model to use the provided tool of the same name.

response_format: ResponseFormat

The format of the response. Only type json_object is currently supported for chat.

โš™๏ธ Request Body

ChatRequest

๐Ÿ”„ Return

ChatResponse

๐ŸŒ Endpoint

/chat post

๐Ÿ”™ Back to Table of Contents


humanloop.chat_deployed

Get a chat response using the project's active deployment.

The active deployment can be a specific model configuration or an experiment.

๐Ÿ› ๏ธ Usage

create_deployed_response = humanloop.chat_deployed(
    messages=[
        {
            "role": "user",
        }
    ],
    project="string_example",
    project_id="string_example",
    session_id="string_example",
    session_reference_id="string_example",
    parent_id="string_example",
    parent_reference_id="string_example",
    inputs={},
    source="string_example",
    metadata={},
    save=True,
    source_datapoint_id="string_example",
    provider_api_keys={},
    num_samples=1,
    stream=False,
    user="string_example",
    seed=1,
    return_inputs=True,
    tool_choice="string_example",
    tool_call="string_example",
    response_format={
        "type": "json_object",
    },
    environment="string_example",
)

โš™๏ธ Parameters

messages: List[ChatMessageWithToolCall]

The messages passed to the to provider chat endpoint.

project: str

Unique project name. If no project exists with this name, a new project will be created.

project_id: str

Unique ID of a project to associate to the log. Either this or project must be provided.

session_id: str

ID of the session to associate the datapoint.

session_reference_id: str

A unique string identifying the session to associate the datapoint to. Allows you to log multiple datapoints to a session (using an ID kept by your internal systems) by passing the same session_reference_id in subsequent log requests. Specify at most one of this or session_id.

parent_id: str

ID associated to the parent datapoint in a session.

parent_reference_id: str

A unique string identifying the previously-logged parent datapoint in a session. Allows you to log nested datapoints with your internal system IDs by passing the same reference ID as parent_id in a prior log request. Specify at most one of this or parent_id. Note that this cannot refer to a datapoint being logged in the same request.

inputs: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

The inputs passed to the prompt template.

source: str

Identifies where the model was called from.

metadata: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

Any additional metadata to record.

save: bool

Whether the request/response payloads will be stored on Humanloop.

source_datapoint_id: str

ID of the source datapoint if this is a log derived from a datapoint in a dataset.

provider_api_keys: ProviderApiKeys

API keys required by each provider to make API calls. The API keys provided here are not stored by Humanloop. If not specified here, Humanloop will fall back to the key saved to your organization.

num_samples: int

The number of generations.

stream: bool

If true, tokens will be sent as data-only server-sent events. If num_samples > 1, samples are streamed back independently.

user: str

End-user ID passed through to provider call.

seed: int

Deprecated field: the seed is instead set as part of the request.config object.

return_inputs: bool

Whether to return the inputs in the response. If false, the response will contain an empty dictionary under inputs. This is useful for reducing the size of the response. Defaults to true.

tool_choice: Union[str, str, str, ToolChoice]

Controls how the model uses tools. The following options are supported: 'none' forces the model to not call a tool; the default when no tools are provided as part of the model config. 'auto' the model can decide to call one of the provided tools; the default when tools are provided as part of the model config. Providing {'type': 'function', 'function': {name': <TOOL_NAME>}} forces the model to use the named function.

tool_call: Union[str, Dict[str, str]]

NB: Deprecated with new tool_choice. Controls how the model uses tools. The following options are supported: 'none' forces the model to not call a tool; the default when no tools are provided as part of the model config. 'auto' the model can decide to call one of the provided tools; the default when tools are provided as part of the model config. Providing {'name': <TOOL_NAME>} forces the model to use the provided tool of the same name.

response_format: ResponseFormat

The format of the response. Only type json_object is currently supported for chat.

environment: str

The environment name used to create a chat response. If not specified, the default environment will be used.

โš™๏ธ Request Body

ChatDeployedRequest

๐Ÿ”„ Return

ChatResponse

๐ŸŒ Endpoint

/chat-deployed post

๐Ÿ”™ Back to Table of Contents


humanloop.chat_experiment

Get a chat response for a specific experiment.

๐Ÿ› ๏ธ Usage

create_experiment_response = humanloop.chat_experiment(
    messages=[
        {
            "role": "user",
        }
    ],
    experiment_id="string_example",
    project="string_example",
    project_id="string_example",
    session_id="string_example",
    session_reference_id="string_example",
    parent_id="string_example",
    parent_reference_id="string_example",
    inputs={},
    source="string_example",
    metadata={},
    save=True,
    source_datapoint_id="string_example",
    provider_api_keys={},
    num_samples=1,
    stream=False,
    user="string_example",
    seed=1,
    return_inputs=True,
    tool_choice="string_example",
    tool_call="string_example",
    response_format={
        "type": "json_object",
    },
)

โš™๏ธ Parameters

messages: List[ChatMessageWithToolCall]

The messages passed to the to provider chat endpoint.

experiment_id: str

If an experiment ID is provided a model configuration will be sampled from the experiments active model configurations.

project: str

Unique project name. If no project exists with this name, a new project will be created.

project_id: str

Unique ID of a project to associate to the log. Either this or project must be provided.

session_id: str

ID of the session to associate the datapoint.

session_reference_id: str

A unique string identifying the session to associate the datapoint to. Allows you to log multiple datapoints to a session (using an ID kept by your internal systems) by passing the same session_reference_id in subsequent log requests. Specify at most one of this or session_id.

parent_id: str

ID associated to the parent datapoint in a session.

parent_reference_id: str

A unique string identifying the previously-logged parent datapoint in a session. Allows you to log nested datapoints with your internal system IDs by passing the same reference ID as parent_id in a prior log request. Specify at most one of this or parent_id. Note that this cannot refer to a datapoint being logged in the same request.

inputs: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

The inputs passed to the prompt template.

source: str

Identifies where the model was called from.

metadata: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

Any additional metadata to record.

save: bool

Whether the request/response payloads will be stored on Humanloop.

source_datapoint_id: str

ID of the source datapoint if this is a log derived from a datapoint in a dataset.

provider_api_keys: ProviderApiKeys

API keys required by each provider to make API calls. The API keys provided here are not stored by Humanloop. If not specified here, Humanloop will fall back to the key saved to your organization.

num_samples: int

The number of chat responses, where each chat response will use a model configuration sampled from the experiment.

stream: bool

If true, tokens will be sent as data-only server-sent events. If num_samples > 1, samples are streamed back independently.

user: str

End-user ID passed through to provider call.

seed: int

Deprecated field: the seed is instead set as part of the request.config object.

return_inputs: bool

Whether to return the inputs in the response. If false, the response will contain an empty dictionary under inputs. This is useful for reducing the size of the response. Defaults to true.

tool_choice: Union[str, str, str, ToolChoice]

Controls how the model uses tools. The following options are supported: 'none' forces the model to not call a tool; the default when no tools are provided as part of the model config. 'auto' the model can decide to call one of the provided tools; the default when tools are provided as part of the model config. Providing {'type': 'function', 'function': {name': <TOOL_NAME>}} forces the model to use the named function.

tool_call: Union[str, Dict[str, str]]

NB: Deprecated with new tool_choice. Controls how the model uses tools. The following options are supported: 'none' forces the model to not call a tool; the default when no tools are provided as part of the model config. 'auto' the model can decide to call one of the provided tools; the default when tools are provided as part of the model config. Providing {'name': <TOOL_NAME>} forces the model to use the provided tool of the same name.

response_format: ResponseFormat

The format of the response. Only type json_object is currently supported for chat.

โš™๏ธ Request Body

ChatExperimentRequest

๐Ÿ”„ Return

ChatResponse

๐ŸŒ Endpoint

/chat-experiment post

๐Ÿ”™ Back to Table of Contents


humanloop.chat_model_config

Get chat response for a specific model configuration.

๐Ÿ› ๏ธ Usage

create_model_config_response = humanloop.chat_model_config(
    messages=[
        {
            "role": "user",
        }
    ],
    model_config_id="string_example",
    project="string_example",
    project_id="string_example",
    session_id="string_example",
    session_reference_id="string_example",
    parent_id="string_example",
    parent_reference_id="string_example",
    inputs={},
    source="string_example",
    metadata={},
    save=True,
    source_datapoint_id="string_example",
    provider_api_keys={},
    num_samples=1,
    stream=False,
    user="string_example",
    seed=1,
    return_inputs=True,
    tool_choice="string_example",
    tool_call="string_example",
    response_format={
        "type": "json_object",
    },
)

โš™๏ธ Parameters

messages: List[ChatMessageWithToolCall]

The messages passed to the to provider chat endpoint.

model_config_id: str

Identifies the model configuration used to create a chat response.

project: str

Unique project name. If no project exists with this name, a new project will be created.

project_id: str

Unique ID of a project to associate to the log. Either this or project must be provided.

session_id: str

ID of the session to associate the datapoint.

session_reference_id: str

A unique string identifying the session to associate the datapoint to. Allows you to log multiple datapoints to a session (using an ID kept by your internal systems) by passing the same session_reference_id in subsequent log requests. Specify at most one of this or session_id.

parent_id: str

ID associated to the parent datapoint in a session.

parent_reference_id: str

A unique string identifying the previously-logged parent datapoint in a session. Allows you to log nested datapoints with your internal system IDs by passing the same reference ID as parent_id in a prior log request. Specify at most one of this or parent_id. Note that this cannot refer to a datapoint being logged in the same request.

inputs: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

The inputs passed to the prompt template.

source: str

Identifies where the model was called from.

metadata: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

Any additional metadata to record.

save: bool

Whether the request/response payloads will be stored on Humanloop.

source_datapoint_id: str

ID of the source datapoint if this is a log derived from a datapoint in a dataset.

provider_api_keys: ProviderApiKeys

API keys required by each provider to make API calls. The API keys provided here are not stored by Humanloop. If not specified here, Humanloop will fall back to the key saved to your organization.

num_samples: int

The number of generations.

stream: bool

If true, tokens will be sent as data-only server-sent events. If num_samples > 1, samples are streamed back independently.

user: str

End-user ID passed through to provider call.

seed: int

Deprecated field: the seed is instead set as part of the request.config object.

return_inputs: bool

Whether to return the inputs in the response. If false, the response will contain an empty dictionary under inputs. This is useful for reducing the size of the response. Defaults to true.

tool_choice: Union[str, str, str, ToolChoice]

Controls how the model uses tools. The following options are supported: 'none' forces the model to not call a tool; the default when no tools are provided as part of the model config. 'auto' the model can decide to call one of the provided tools; the default when tools are provided as part of the model config. Providing {'type': 'function', 'function': {name': <TOOL_NAME>}} forces the model to use the named function.

tool_call: Union[str, Dict[str, str]]

NB: Deprecated with new tool_choice. Controls how the model uses tools. The following options are supported: 'none' forces the model to not call a tool; the default when no tools are provided as part of the model config. 'auto' the model can decide to call one of the provided tools; the default when tools are provided as part of the model config. Providing {'name': <TOOL_NAME>} forces the model to use the provided tool of the same name.

response_format: ResponseFormat

The format of the response. Only type json_object is currently supported for chat.

โš™๏ธ Request Body

ChatModelConfigRequest

๐Ÿ”„ Return

ChatResponse

๐ŸŒ Endpoint

/chat-model-config post

๐Ÿ”™ Back to Table of Contents


humanloop.complete

Create a completion by providing details of the model configuration in the request.

๐Ÿ› ๏ธ Usage

create_response = humanloop.complete(
    model_config={
        "provider": "openai",
        "model": "model_example",
        "max_tokens": -1,
        "temperature": 1,
        "top_p": 1,
        "presence_penalty": 0,
        "frequency_penalty": 0,
        "endpoint": "complete",
        "prompt_template": "{{question}}",
    },
    project="string_example",
    project_id="string_example",
    session_id="string_example",
    session_reference_id="string_example",
    parent_id="string_example",
    parent_reference_id="string_example",
    inputs={},
    source="string_example",
    metadata={},
    save=True,
    source_datapoint_id="string_example",
    provider_api_keys={},
    num_samples=1,
    stream=False,
    user="string_example",
    seed=1,
    return_inputs=True,
    logprobs=1,
    suffix="string_example",
)

โš™๏ธ Parameters

model_config: ModelConfigCompletionRequest

The model configuration used to generate.

project: str

Unique project name. If no project exists with this name, a new project will be created.

project_id: str

Unique ID of a project to associate to the log. Either this or project must be provided.

session_id: str

ID of the session to associate the datapoint.

session_reference_id: str

A unique string identifying the session to associate the datapoint to. Allows you to log multiple datapoints to a session (using an ID kept by your internal systems) by passing the same session_reference_id in subsequent log requests. Specify at most one of this or session_id.

parent_id: str

ID associated to the parent datapoint in a session.

parent_reference_id: str

A unique string identifying the previously-logged parent datapoint in a session. Allows you to log nested datapoints with your internal system IDs by passing the same reference ID as parent_id in a prior log request. Specify at most one of this or parent_id. Note that this cannot refer to a datapoint being logged in the same request.

inputs: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

The inputs passed to the prompt template.

source: str

Identifies where the model was called from.

metadata: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

Any additional metadata to record.

save: bool

Whether the request/response payloads will be stored on Humanloop.

source_datapoint_id: str

ID of the source datapoint if this is a log derived from a datapoint in a dataset.

provider_api_keys: ProviderApiKeys

API keys required by each provider to make API calls. The API keys provided here are not stored by Humanloop. If not specified here, Humanloop will fall back to the key saved to your organization.

num_samples: int

The number of generations.

stream: bool

If true, tokens will be sent as data-only server-sent events. If num_samples > 1, samples are streamed back independently.

user: str

End-user ID passed through to provider call.

seed: int

Deprecated field: the seed is instead set as part of the request.config object.

return_inputs: bool

Whether to return the inputs in the response. If false, the response will contain an empty dictionary under inputs. This is useful for reducing the size of the response. Defaults to true.

logprobs: int

Include the log probabilities of the top n tokens in the provider_response

suffix: str

The suffix that comes after a completion of inserted text. Useful for completions that act like inserts.

โš™๏ธ Request Body

CompletionRequest

๐Ÿ”„ Return

CompletionResponse

๐ŸŒ Endpoint

/completion post

๐Ÿ”™ Back to Table of Contents


humanloop.complete_deployed

Create a completion using the project's active deployment.

The active deployment can be a specific model configuration or an experiment.

๐Ÿ› ๏ธ Usage

create_deployed_response = humanloop.complete_deployed(
    project="string_example",
    project_id="string_example",
    session_id="string_example",
    session_reference_id="string_example",
    parent_id="string_example",
    parent_reference_id="string_example",
    inputs={},
    source="string_example",
    metadata={},
    save=True,
    source_datapoint_id="string_example",
    provider_api_keys={},
    num_samples=1,
    stream=False,
    user="string_example",
    seed=1,
    return_inputs=True,
    logprobs=1,
    suffix="string_example",
    environment="string_example",
)

โš™๏ธ Parameters

project: str

Unique project name. If no project exists with this name, a new project will be created.

project_id: str

Unique ID of a project to associate to the log. Either this or project must be provided.

session_id: str

ID of the session to associate the datapoint.

session_reference_id: str

A unique string identifying the session to associate the datapoint to. Allows you to log multiple datapoints to a session (using an ID kept by your internal systems) by passing the same session_reference_id in subsequent log requests. Specify at most one of this or session_id.

parent_id: str

ID associated to the parent datapoint in a session.

parent_reference_id: str

A unique string identifying the previously-logged parent datapoint in a session. Allows you to log nested datapoints with your internal system IDs by passing the same reference ID as parent_id in a prior log request. Specify at most one of this or parent_id. Note that this cannot refer to a datapoint being logged in the same request.

inputs: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

The inputs passed to the prompt template.

source: str

Identifies where the model was called from.

metadata: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

Any additional metadata to record.

save: bool

Whether the request/response payloads will be stored on Humanloop.

source_datapoint_id: str

ID of the source datapoint if this is a log derived from a datapoint in a dataset.

provider_api_keys: ProviderApiKeys

API keys required by each provider to make API calls. The API keys provided here are not stored by Humanloop. If not specified here, Humanloop will fall back to the key saved to your organization.

num_samples: int

The number of generations.

stream: bool

If true, tokens will be sent as data-only server-sent events. If num_samples > 1, samples are streamed back independently.

user: str

End-user ID passed through to provider call.

seed: int

Deprecated field: the seed is instead set as part of the request.config object.

return_inputs: bool

Whether to return the inputs in the response. If false, the response will contain an empty dictionary under inputs. This is useful for reducing the size of the response. Defaults to true.

logprobs: int

Include the log probabilities of the top n tokens in the provider_response

suffix: str

The suffix that comes after a completion of inserted text. Useful for completions that act like inserts.

environment: str

The environment name used to create a chat response. If not specified, the default environment will be used.

โš™๏ธ Request Body

CompletionDeployedRequest

๐Ÿ”„ Return

CompletionResponse

๐ŸŒ Endpoint

/completion-deployed post

๐Ÿ”™ Back to Table of Contents


humanloop.complete_experiment

Create a completion for a specific experiment.

๐Ÿ› ๏ธ Usage

create_experiment_response = humanloop.complete_experiment(
    experiment_id="string_example",
    project="string_example",
    project_id="string_example",
    session_id="string_example",
    session_reference_id="string_example",
    parent_id="string_example",
    parent_reference_id="string_example",
    inputs={},
    source="string_example",
    metadata={},
    save=True,
    source_datapoint_id="string_example",
    provider_api_keys={},
    num_samples=1,
    stream=False,
    user="string_example",
    seed=1,
    return_inputs=True,
    logprobs=1,
    suffix="string_example",
)

โš™๏ธ Parameters

experiment_id: str

If an experiment ID is provided a model configuration will be sampled from the experiments active model configurations.

project: str

Unique project name. If no project exists with this name, a new project will be created.

project_id: str

Unique ID of a project to associate to the log. Either this or project must be provided.

session_id: str

ID of the session to associate the datapoint.

session_reference_id: str

A unique string identifying the session to associate the datapoint to. Allows you to log multiple datapoints to a session (using an ID kept by your internal systems) by passing the same session_reference_id in subsequent log requests. Specify at most one of this or session_id.

parent_id: str

ID associated to the parent datapoint in a session.

parent_reference_id: str

A unique string identifying the previously-logged parent datapoint in a session. Allows you to log nested datapoints with your internal system IDs by passing the same reference ID as parent_id in a prior log request. Specify at most one of this or parent_id. Note that this cannot refer to a datapoint being logged in the same request.

inputs: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

The inputs passed to the prompt template.

source: str

Identifies where the model was called from.

metadata: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

Any additional metadata to record.

save: bool

Whether the request/response payloads will be stored on Humanloop.

source_datapoint_id: str

ID of the source datapoint if this is a log derived from a datapoint in a dataset.

provider_api_keys: ProviderApiKeys

API keys required by each provider to make API calls. The API keys provided here are not stored by Humanloop. If not specified here, Humanloop will fall back to the key saved to your organization.

num_samples: int

The number of chat responses, where each chat response will use a model configuration sampled from the experiment.

stream: bool

If true, tokens will be sent as data-only server-sent events. If num_samples > 1, samples are streamed back independently.

user: str

End-user ID passed through to provider call.

seed: int

Deprecated field: the seed is instead set as part of the request.config object.

return_inputs: bool

Whether to return the inputs in the response. If false, the response will contain an empty dictionary under inputs. This is useful for reducing the size of the response. Defaults to true.

logprobs: int

Include the log probabilities of the top n tokens in the provider_response

suffix: str

The suffix that comes after a completion of inserted text. Useful for completions that act like inserts.

โš™๏ธ Request Body

CompletionExperimentRequest

๐Ÿ”„ Return

CompletionResponse

๐ŸŒ Endpoint

/completion-experiment post

๐Ÿ”™ Back to Table of Contents


humanloop.complete_model_configuration

Create a completion for a specific model configuration.

๐Ÿ› ๏ธ Usage

create_model_config_response = humanloop.complete_model_configuration(
    model_config_id="string_example",
    project="string_example",
    project_id="string_example",
    session_id="string_example",
    session_reference_id="string_example",
    parent_id="string_example",
    parent_reference_id="string_example",
    inputs={},
    source="string_example",
    metadata={},
    save=True,
    source_datapoint_id="string_example",
    provider_api_keys={},
    num_samples=1,
    stream=False,
    user="string_example",
    seed=1,
    return_inputs=True,
    logprobs=1,
    suffix="string_example",
)

โš™๏ธ Parameters

model_config_id: str

Identifies the model configuration used to create a chat response.

project: str

Unique project name. If no project exists with this name, a new project will be created.

project_id: str

Unique ID of a project to associate to the log. Either this or project must be provided.

session_id: str

ID of the session to associate the datapoint.

session_reference_id: str

A unique string identifying the session to associate the datapoint to. Allows you to log multiple datapoints to a session (using an ID kept by your internal systems) by passing the same session_reference_id in subsequent log requests. Specify at most one of this or session_id.

parent_id: str

ID associated to the parent datapoint in a session.

parent_reference_id: str

A unique string identifying the previously-logged parent datapoint in a session. Allows you to log nested datapoints with your internal system IDs by passing the same reference ID as parent_id in a prior log request. Specify at most one of this or parent_id. Note that this cannot refer to a datapoint being logged in the same request.

inputs: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

The inputs passed to the prompt template.

source: str

Identifies where the model was called from.

metadata: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

Any additional metadata to record.

save: bool

Whether the request/response payloads will be stored on Humanloop.

source_datapoint_id: str

ID of the source datapoint if this is a log derived from a datapoint in a dataset.

provider_api_keys: ProviderApiKeys

API keys required by each provider to make API calls. The API keys provided here are not stored by Humanloop. If not specified here, Humanloop will fall back to the key saved to your organization.

num_samples: int

The number of generations.

stream: bool

If true, tokens will be sent as data-only server-sent events. If num_samples > 1, samples are streamed back independently.

user: str

End-user ID passed through to provider call.

seed: int

Deprecated field: the seed is instead set as part of the request.config object.

return_inputs: bool

Whether to return the inputs in the response. If false, the response will contain an empty dictionary under inputs. This is useful for reducing the size of the response. Defaults to true.

logprobs: int

Include the log probabilities of the top n tokens in the provider_response

suffix: str

The suffix that comes after a completion of inserted text. Useful for completions that act like inserts.

โš™๏ธ Request Body

CompletionModelConfigRequest

๐Ÿ”„ Return

CompletionResponse

๐ŸŒ Endpoint

/completion-model-config post

๐Ÿ”™ Back to Table of Contents


humanloop.datapoints.delete

Deprecated

Delete a list of datapoints by their IDs.

WARNING: This endpoint has been decommisioned and no longer works. Please use the v5 datasets API instead.

๐Ÿ› ๏ธ Usage

humanloop.datapoints.delete()

๐ŸŒ Endpoint

/datapoints delete

๐Ÿ”™ Back to Table of Contents


humanloop.datapoints.get

Get a datapoint by ID.

๐Ÿ› ๏ธ Usage

get_response = humanloop.datapoints.get(
    id="id_example",
)

โš™๏ธ Parameters

id: str

String ID of datapoint.

๐Ÿ”„ Return

DatapointResponse

๐ŸŒ Endpoint

/datapoints/{id} get

๐Ÿ”™ Back to Table of Contents


humanloop.datapoints.update

Deprecated

Edit the input, messages and criteria fields of a datapoint.

WARNING: This endpoint has been decommisioned and no longer works. Please use the v5 datasets API instead.

๐Ÿ› ๏ธ Usage

update_response = humanloop.datapoints.update(
    id="id_example",
)

โš™๏ธ Parameters

id: str

String ID of datapoint.

๐Ÿ”„ Return

DatapointResponse

๐ŸŒ Endpoint

/datapoints/{id} patch

๐Ÿ”™ Back to Table of Contents


humanloop.datasets.create

Create a new dataset for a project.

๐Ÿ› ๏ธ Usage

create_response = humanloop.datasets.create(
    description="string_example",
    name="string_example",
    project_id="project_id_example",
)

โš™๏ธ Parameters

description: str

The description of the dataset.

name: str

The name of the dataset.

project_id: str

โš™๏ธ Request Body

CreateDatasetRequest

๐Ÿ”„ Return

DatasetResponse

๐ŸŒ Endpoint

/projects/{project_id}/datasets post

๐Ÿ”™ Back to Table of Contents


humanloop.datasets.create_datapoint

Create a new datapoint for a dataset.

Here in the v4 API, this has the following behaviour:

  • Retrieve the current latest version of the dataset.
  • Construct a new version of the dataset with the new testcases added.
  • Store that latest version as a committed version with an autogenerated commit message and return the new datapoints

๐Ÿ› ๏ธ Usage

create_datapoint_response = humanloop.datasets.create_datapoint(
    body={
        "log_ids": ["log_ids_example"],
    },
    dataset_id="dataset_id_example",
    log_ids=["string_example"],
    inputs={
        "key": "string_example",
    },
    messages=[
        {
            "role": "user",
        }
    ],
    target={
        "key": "string_example",
    },
)

โš™๏ธ Parameters

dataset_id: str

String ID of dataset. Starts with evts_.

requestBody: DatasetsCreateDatapointRequest

๐Ÿ”„ Return

DatasetsCreateDatapointResponse

๐ŸŒ Endpoint

/datasets/{dataset_id}/datapoints post

๐Ÿ”™ Back to Table of Contents


humanloop.datasets.delete

Delete a dataset by ID.

๐Ÿ› ๏ธ Usage

delete_response = humanloop.datasets.delete(
    id="id_example",
)

โš™๏ธ Parameters

id: str

String ID of dataset. Starts with evts_.

๐ŸŒ Endpoint

/datasets/{id} delete

๐Ÿ”™ Back to Table of Contents


humanloop.datasets.get

Get a single dataset by ID.

๐Ÿ› ๏ธ Usage

get_response = humanloop.datasets.get(
    id="id_example",
)

โš™๏ธ Parameters

id: str

String ID of dataset. Starts with evts_.

๐Ÿ”„ Return

DatasetResponse

๐ŸŒ Endpoint

/datasets/{id} get

๐Ÿ”™ Back to Table of Contents


humanloop.datasets.list

Get all Datasets for an organization.

๐Ÿ› ๏ธ Usage

list_response = humanloop.datasets.list()

๐Ÿ”„ Return

DatasetsListResponse

๐ŸŒ Endpoint

/datasets get

๐Ÿ”™ Back to Table of Contents


humanloop.datasets.list_all_for_project

Deprecated

Get all datasets for a project.

๐Ÿ› ๏ธ Usage

list_all_for_project_response = humanloop.datasets.list_all_for_project(
    project_id="project_id_example",
)

โš™๏ธ Parameters

project_id: str

๐Ÿ”„ Return

DatasetsListAllForProjectResponse

๐ŸŒ Endpoint

/projects/{project_id}/datasets get

๐Ÿ”™ Back to Table of Contents


humanloop.datasets.list_datapoints

Get datapoints for a dataset.

๐Ÿ› ๏ธ Usage

list_datapoints_response = humanloop.datasets.list_datapoints(
    dataset_id="dataset_id_example",
    page=0,
    size=50,
)

โš™๏ธ Parameters

dataset_id: str

String ID of dataset. Starts with evts_.

page: int
size: int

๐Ÿ”„ Return

PaginatedDataDatapointResponse

๐ŸŒ Endpoint

/datasets/{dataset_id}/datapoints get

๐Ÿ”™ Back to Table of Contents


humanloop.datasets.update

Update a testset by ID.

๐Ÿ› ๏ธ Usage

update_response = humanloop.datasets.update(
    id="id_example",
    description="string_example",
    name="string_example",
)

โš™๏ธ Parameters

id: str

String ID of testset. Starts with evts_.

description: str

The description of the dataset.

name: str

The name of the dataset.

โš™๏ธ Request Body

UpdateDatasetRequest

๐Ÿ”„ Return

DatasetResponse

๐ŸŒ Endpoint

/datasets/{id} patch

๐Ÿ”™ Back to Table of Contents


humanloop.evaluations.add_evaluators

Add evaluators to an existing evaluation run.

๐Ÿ› ๏ธ Usage

add_evaluators_response = humanloop.evaluations.add_evaluators(
    id="id_example",
    evaluator_ids=["string_example"],
    evaluator_version_ids=["string_example"],
)

โš™๏ธ Parameters

id: str

String ID of evaluation run. Starts with ev_.

evaluator_ids: AddEvaluatorsRequestEvaluatorIds
evaluator_version_ids: AddEvaluatorsRequestEvaluatorVersionIds

โš™๏ธ Request Body

AddEvaluatorsRequest

๐Ÿ”„ Return

EvaluationResponse

๐ŸŒ Endpoint

/evaluations/{id}/evaluators patch

๐Ÿ”™ Back to Table of Contents


humanloop.evaluations.create

Create an evaluation.

๐Ÿ› ๏ธ Usage

create_response = humanloop.evaluations.create(
    config_id="string_example",
    evaluator_ids=["string_example"],
    dataset_id="string_example",
    project_id="project_id_example",
    provider_api_keys={},
    hl_generated=True,
)

โš™๏ธ Parameters

config_id: str

ID of the config to evaluate. Starts with config_.

evaluator_ids: CreateEvaluationRequestEvaluatorIds
dataset_id: str

ID of the dataset to use in this evaluation. Starts with evts_.

project_id: str

String ID of project. Starts with pr_.

provider_api_keys: ProviderApiKeys

API keys required by each provider to make API calls. The API keys provided here are not stored by Humanloop. If not specified here, Humanloop will fall back to the key saved to your organization. Ensure you provide an API key for the provider for the model config you are evaluating, or have one saved to your organization.

hl_generated: bool

Whether the log generations for this evaluation should be performed by Humanloop. If False, the log generations should be submitted by the user via the API.

โš™๏ธ Request Body

CreateEvaluationRequest

๐Ÿ”„ Return

EvaluationResponse

๐ŸŒ Endpoint

/projects/{project_id}/evaluations post

๐Ÿ”™ Back to Table of Contents


humanloop.evaluations.get

Get evaluation by ID.

๐Ÿ› ๏ธ Usage

get_response = humanloop.evaluations.get(
    id="id_example",
    evaluator_aggregates=True,
    evaluatee_id="string_example",
)

โš™๏ธ Parameters

id: str

String ID of evaluation run. Starts with ev_.

evaluator_aggregates: bool

Whether to include evaluator aggregates in the response.

evaluatee_id: str

String ID of evaluatee version to return. If not defined, the first evaluatee will be returned. Starts with evv_.

๐Ÿ”„ Return

EvaluationResponse

๐ŸŒ Endpoint

/evaluations/{id} get

๐Ÿ”™ Back to Table of Contents


humanloop.evaluations.list

Get the evaluations associated with a project.

Sorting and filtering are supported through query params for categorical columns and the created_at timestamp.

Sorting is supported for the dataset, config, status and evaluator-{evaluator_id} columns. Specify sorting with the sort query param, with values {column}.{ordering}. E.g. ?sort=dataset.asc&sort=status.desc will yield a multi-column sort. First by dataset then by status.

Filtering is supported for the id, dataset, config and status columns.

Specify filtering with the id_filter, dataset_filter, config_filter and status_filter query params.

E.g. ?dataset_filter=my_dataset&dataset_filter=my_other_dataset&status_filter=running will only show rows where the dataset is "my_dataset" or "my_other_dataset", and where the status is "running".

An additional date range filter is supported for the created_at column. Use the start_date and end_date query parameters to configure this.

๐Ÿ› ๏ธ Usage

list_response = humanloop.evaluations.list(
    project_id="project_id_example",
    id=["string_example"],
    start_date="1970-01-01",
    end_date="1970-01-01",
    size=50,
    page=0,
    evaluatee_id="string_example",
)

โš™๏ธ Parameters

project_id: str

String ID of project. Starts with pr_.

id: List[str]

A list of evaluation run ids to filter on. Starts with ev_.

start_date: date

Only return evaluations created after this date.

end_date: date

Only return evaluations created before this date.

size: int
page: int
evaluatee_id: str

String ID of evaluatee version to return. If not defined, the first evaluatee will be returned. Starts with evv_.

๐Ÿ”„ Return

PaginatedDataEvaluationResponse

๐ŸŒ Endpoint

/evaluations get

๐Ÿ”™ Back to Table of Contents


humanloop.evaluations.list_all_for_project

Deprecated

Get all the evaluations associated with your project.

Deprecated: This is a legacy unpaginated endpoint. Use /evaluations instead, with appropriate sorting, filtering and pagination options.

๐Ÿ› ๏ธ Usage

list_all_for_project_response = humanloop.evaluations.list_all_for_project(
    project_id="project_id_example",
    evaluatee_id="string_example",
    evaluator_aggregates=True,
)

โš™๏ธ Parameters

project_id: str

String ID of project. Starts with pr_.

evaluatee_id: str

String ID of evaluatee version to return. If not defined, the first evaluatee will be returned. Starts with evv_.

evaluator_aggregates: bool

Whether to include evaluator aggregates in the response.

๐Ÿ”„ Return

EvaluationsGetForProjectResponse

๐ŸŒ Endpoint

/projects/{project_id}/evaluations get

๐Ÿ”™ Back to Table of Contents


humanloop.evaluations.list_datapoints

Get testcases by evaluation ID.

๐Ÿ› ๏ธ Usage

list_datapoints_response = humanloop.evaluations.list_datapoints(
    id="id_example",
    page=1,
    size=10,
    evaluatee_id="string_example",
)

โš™๏ธ Parameters

id: str

String ID of evaluation. Starts with ev_.

page: int

Page to fetch. Starts from 1.

size: int

Number of evaluation results to retrieve.

evaluatee_id: str

String ID of evaluatee version to return. If not defined, the first evaluatee will be returned. Starts with evv_.

๐Ÿ”„ Return

PaginatedDataEvaluationDatapointSnapshotResponse

๐ŸŒ Endpoint

/evaluations/{id}/datapoints get

๐Ÿ”™ Back to Table of Contents


humanloop.evaluations.log

Log an external generation to an evaluation run for a datapoint.

The run must have status 'running'.

๐Ÿ› ๏ธ Usage

log_response = humanloop.evaluations.log(
    datapoint_id="string_example",
    log={
        "save": True,
    },
    evaluation_id="evaluation_id_example",
    evaluatee_id="string_example",
)

โš™๏ธ Parameters

datapoint_id: str

The datapoint for which a log was generated. Must be one of the datapoints in the dataset being evaluated.

log: LogRequest

The log generated for the datapoint.

evaluation_id: str

ID of the evaluation run. Starts with evrun_.

evaluatee_id: str

String ID of evaluatee version to return. If not defined, the first evaluatee will be returned. Starts with evv_.

โš™๏ธ Request Body

CreateEvaluationLogRequest

๐Ÿ”„ Return

CreateLogResponse

๐ŸŒ Endpoint

/evaluations/{evaluation_id}/log post

๐Ÿ”™ Back to Table of Contents


humanloop.evaluations.result

Log an evaluation result to an evaluation run.

The run must have status 'running'. One of result or error must be provided.

๐Ÿ› ๏ธ Usage

result_response = humanloop.evaluations.result(
    log_id="string_example",
    evaluator_id="string_example",
    evaluation_id="evaluation_id_example",
    result=True,
    error="string_example",
    evaluatee_id="string_example",
)

โš™๏ธ Parameters

log_id: str

The log that was evaluated. Must have as its source_datapoint_id one of the datapoints in the dataset being evaluated.

evaluator_id: str

ID of the evaluator that evaluated the log. Starts with evfn_. Must be one of the evaluator IDs associated with the evaluation run being logged to.

evaluation_id: str

ID of the evaluation run. Starts with evrun_.

result: Union[bool, int, Union[int, float]]

The result value of the evaluation.

error: str

An error that occurred during evaluation.

evaluatee_id: str

String ID of evaluatee version to return. If not defined, the first evaluatee will be returned. Starts with evv_.

โš™๏ธ Request Body

CreateEvaluationResultLogRequest

๐Ÿ”„ Return

EvaluationResultResponse

๐ŸŒ Endpoint

/evaluations/{evaluation_id}/result post

๐Ÿ”™ Back to Table of Contents


humanloop.evaluations.update_status

Update the status of an evaluation run.

Can only be used to update the status of an evaluation run that uses external or human evaluators. The evaluation must currently have status 'running' if swithcing to completed, or it must have status 'completed' if switching back to 'running'.

๐Ÿ› ๏ธ Usage

update_status_response = humanloop.evaluations.update_status(
    status="pending",
    id="id_example",
)

โš™๏ธ Parameters

status: EvaluationStatus

The new status of the evaluation.

id: str

String ID of evaluation run. Starts with ev_.

โš™๏ธ Request Body

UpdateEvaluationStatusRequest

๐Ÿ”„ Return

EvaluationResponse

๐ŸŒ Endpoint

/evaluations/{id}/status patch

๐Ÿ”™ Back to Table of Contents


humanloop.evaluators.create

Create an evaluator within your organization.

๐Ÿ› ๏ธ Usage

create_response = humanloop.evaluators.create(
    description="string_example",
    name="a",
    arguments_type="target_free",
    return_type="boolean",
    type="python",
    code="string_example",
    model_config={
        "provider": "openai",
        "model": "model_example",
        "max_tokens": -1,
        "temperature": 1,
        "top_p": 1,
        "presence_penalty": 0,
        "frequency_penalty": 0,
        "endpoint": "complete",
        "prompt_template": "{{question}}",
    },
)

โš™๏ธ Parameters

description: str

The description of the evaluator.

name: str

The name of the evaluator.

arguments_type: EvaluatorArgumentsType

Whether this evaluator is target-free or target-required.

return_type: EvaluatorReturnTypeEnum

The type of the return value of the evaluator.

type: EvaluatorType

The type of the evaluator.

code: str

The code for the evaluator. This code will be executed in a sandboxed environment.

model_config: ModelConfigCompletionRequest

The model configuration used to generate.

โš™๏ธ Request Body

CreateEvaluatorRequest

๐Ÿ”„ Return

EvaluatorResponse

๐ŸŒ Endpoint

/evaluators post

๐Ÿ”™ Back to Table of Contents


humanloop.evaluators.delete

Delete an evaluator within your organization.

๐Ÿ› ๏ธ Usage

humanloop.evaluators.delete(
    id="id_example",
)

โš™๏ธ Parameters

id: str

๐ŸŒ Endpoint

/evaluators/{id} delete

๐Ÿ”™ Back to Table of Contents


humanloop.evaluators.get

Get an evaluator within your organization.

๐Ÿ› ๏ธ Usage

get_response = humanloop.evaluators.get(
    id="id_example",
)

โš™๏ธ Parameters

id: str

๐Ÿ”„ Return

EvaluatorResponse

๐ŸŒ Endpoint

/evaluators/{id} get

๐Ÿ”™ Back to Table of Contents


humanloop.evaluators.list

Get all evaluators within your organization.

๐Ÿ› ๏ธ Usage

list_response = humanloop.evaluators.list()

๐Ÿ”„ Return

EvaluatorsListResponse

๐ŸŒ Endpoint

/evaluators get

๐Ÿ”™ Back to Table of Contents


humanloop.evaluators.update

Update an evaluator within your organization.

๐Ÿ› ๏ธ Usage

update_response = humanloop.evaluators.update(
    id="id_example",
    description="string_example",
    name="string_example",
    arguments_type="target_free",
    return_type="boolean",
    code="string_example",
    model_config={
        "provider": "openai",
        "model": "model_example",
        "max_tokens": -1,
        "temperature": 1,
        "top_p": 1,
        "presence_penalty": 0,
        "frequency_penalty": 0,
        "endpoint": "complete",
        "prompt_template": "{{question}}",
    },
)

โš™๏ธ Parameters

id: str
description: str

The description of the evaluator.

name: str

The name of the evaluator.

arguments_type: EvaluatorArgumentsType

Whether this evaluator is target-free or target-required.

return_type: EvaluatorReturnTypeEnum

The type of the return value of the evaluator.

code: str

The code for the evaluator. This code will be executed in a sandboxed environment.

model_config: ModelConfigCompletionRequest

The model configuration used to generate.

โš™๏ธ Request Body

UpdateEvaluatorRequest

๐Ÿ”„ Return

EvaluatorResponse

๐ŸŒ Endpoint

/evaluators/{id} patch

๐Ÿ”™ Back to Table of Contents


humanloop.experiments.create

Create an experiment for your project.

You can optionally specify IDs of your project's model configs to include in the experiment, along with a set of labels to consider as positive feedback and whether the experiment should be set as active.

๐Ÿ› ๏ธ Usage

create_response = humanloop.experiments.create(
    name="string_example",
    positive_labels=[
        {
            "type": "type_example",
            "value": "value_example",
        }
    ],
    project_id="project_id_example",
    config_ids=["string_example"],
    set_active=False,
)

โš™๏ธ Parameters

name: str

Name of experiment.

positive_labels: List[PositiveLabel]

Feedback labels to treat as positive user feedback. Used to monitor the performance of model configs in the experiment.

project_id: str

String ID of project. Starts with pr_.

config_ids: CreateExperimentRequestConfigIds
set_active: bool

Whether to set the created project as the project's active experiment.

โš™๏ธ Request Body

CreateExperimentRequest

๐Ÿ”„ Return

ExperimentResponse

๐ŸŒ Endpoint

/projects/{project_id}/experiments post

๐Ÿ”™ Back to Table of Contents


humanloop.experiments.delete

Delete the experiment with the specified ID.

๐Ÿ› ๏ธ Usage

humanloop.experiments.delete(
    experiment_id="experiment_id_example",
)

โš™๏ธ Parameters

experiment_id: str

String ID of experiment. Starts with exp_.

๐ŸŒ Endpoint

/experiments/{experiment_id} delete

๐Ÿ”™ Back to Table of Contents


humanloop.experiments.list

Get an array of experiments associated to your project.

๐Ÿ› ๏ธ Usage

list_response = humanloop.experiments.list(
    project_id="project_id_example",
)

โš™๏ธ Parameters

project_id: str

String ID of project. Starts with pr_.

๐Ÿ”„ Return

ExperimentsListResponse

๐ŸŒ Endpoint

/projects/{project_id}/experiments get

๐Ÿ”™ Back to Table of Contents


humanloop.experiments.sample

Samples a model config from the experiment's active model configs.

๐Ÿ› ๏ธ Usage

sample_response = humanloop.experiments.sample(
    experiment_id="experiment_id_example",
)

โš™๏ธ Parameters

experiment_id: str

String ID of experiment. Starts with exp_.

๐Ÿ”„ Return

GetModelConfigResponse

๐ŸŒ Endpoint

/experiments/{experiment_id}/model-config get

๐Ÿ”™ Back to Table of Contents


humanloop.experiments.update

Update your experiment, including registering and de-registering model configs.

๐Ÿ› ๏ธ Usage

update_response = humanloop.experiments.update(
    experiment_id="experiment_id_example",
    name="string_example",
    positive_labels=[
        {
            "type": "type_example",
            "value": "value_example",
        }
    ],
    config_ids_to_register=["string_example"],
    config_ids_to_deregister=["string_example"],
)

โš™๏ธ Parameters

experiment_id: str

String ID of experiment. Starts with exp_.

name: str

Name of experiment.

positive_labels: List[PositiveLabel]

Feedback labels to treat as positive user feedback. Used to monitor the performance of model configs in the experiment.

config_ids_to_register: UpdateExperimentRequestConfigIdsToRegister
config_ids_to_deregister: UpdateExperimentRequestConfigIdsToDeregister

โš™๏ธ Request Body

UpdateExperimentRequest

๐Ÿ”„ Return

ExperimentResponse

๐ŸŒ Endpoint

/experiments/{experiment_id} patch

๐Ÿ”™ Back to Table of Contents


humanloop.feedback

Submit an array of feedback for existing data_ids

๐Ÿ› ๏ธ Usage

feedback_response = humanloop.feedback(
    body=[
        {
            "type": "string_example",
        }
    ],
    type="string_example",
    value="string_example",
    data_id="string_example",
    user="string_example",
    created_at="1970-01-01T00:00:00.00Z",
    unset=True,
)

โš™๏ธ Parameters

type: Union[FeedbackType, str]

The type of feedback. The default feedback types available are 'rating', 'action', 'issue', 'correction', and 'comment'.

value: str

The feedback value to be set. This field should be left blank when unsetting 'rating', 'correction' or 'comment', but is required otherwise.

data_id: str

ID to associate the feedback to a previously logged datapoint.

user: str

A unique identifier to who provided the feedback.

created_at: datetime

User defined timestamp for when the feedback was created.

unset: bool

If true, the value for this feedback is unset.

โš™๏ธ Request Body

FeedbackSubmitRequest

๐Ÿ”„ Return

FeedbackSubmitResponse

๐ŸŒ Endpoint

/feedback post

๐Ÿ”™ Back to Table of Contents


humanloop.logs.delete

Delete

๐Ÿ› ๏ธ Usage

humanloop.logs.delete(
    id=["string_example"],
)

โš™๏ธ Parameters

id: List[str]

๐ŸŒ Endpoint

/logs delete

๐Ÿ”™ Back to Table of Contents


humanloop.logs.get

Retrieve a log by log id.

๐Ÿ› ๏ธ Usage

get_response = humanloop.logs.get(
    id="id_example",
)

โš™๏ธ Parameters

id: str

String ID of log to return. Starts with data_.

๐Ÿ”„ Return

LogResponse

๐ŸŒ Endpoint

/logs/{id} get

๐Ÿ”™ Back to Table of Contents


humanloop.logs.list

Retrieve paginated logs from the server.

Sorting and filtering are supported through query params.

Sorting is supported for the source, model, timestamp, and feedback-{output_name} columns. Specify sorting with the sort query param, with values {column}.{ordering}. E.g. ?sort=source.asc&sort=model.desc will yield a multi-column sort. First by source then by model.

Filtering is supported for the source, model, feedback-{output_name}, evaluator-{evaluator_external_id} columns.

Specify filtering with the source_filter, model_filter, feedback-{output.name}_filter and evaluator-{evaluator_external_id}_filter query params.

E.g. ?source_filter=AI&source_filter=user_1234&feedback-explicit_filter=good will only show rows where the source is "AI" or "user_1234", and where the latest feedback for the "explicit" output group is "good".

An additional date range filter is supported for the Timestamp column (i.e. Log.created_at). These are supported through the start_date and end_date query parameters. The date format could be either date: YYYY-MM-DD, e.g. 2024-01-01 or datetime: YYYY-MM-DD[T]HH:MM[:SS[.ffffff]][Z or [ยฑ]HH[:]MM], e.g. 2024-01-01T00:00:00Z.

Searching is supported for the model inputs and output. Specify a search term with the search query param. E.g. ?search=hello%20there will cause a case-insensitive search across model inputs and output.

๐Ÿ› ๏ธ Usage

list_response = humanloop.logs.list(
    project_id="project_id_example",
    search="string_example",
    metadata_search="string_example",
    version_status="uncommitted",
    start_date="1970-01-01",
    end_date="1970-01-01",
    size=50,
    page=0,
)

โš™๏ธ Parameters

project_id: str
search: str
metadata_search: str
version_status: VersionStatus
start_date: Union[date, datetime]
end_date: Union[date, datetime]
size: int
page: int

๐Ÿ”„ Return

PaginatedDataLogResponse

๐ŸŒ Endpoint

/logs get

๐Ÿ”™ Back to Table of Contents


humanloop.log

Log a datapoint or array of datapoints to your Humanloop project.

๐Ÿ› ๏ธ Usage

log_response = humanloop.log(
    body=[
        {
            "save": True,
        }
    ],
    project="string_example",
    project_id="string_example",
    session_id="string_example",
    session_reference_id="string_example",
    parent_id="string_example",
    parent_reference_id="string_example",
    inputs={},
    source="string_example",
    metadata={},
    save=True,
    source_datapoint_id="string_example",
    reference_id="string_example",
    trial_id="string_example",
    messages=[
        {
            "role": "user",
        }
    ],
    output="string_example",
    judgment=True,
    config_id="string_example",
    config={
        "provider": "openai",
        "model": "model_example",
        "max_tokens": -1,
        "temperature": 1,
        "top_p": 1,
        "presence_penalty": 0,
        "frequency_penalty": 0,
        "endpoint": "complete",
        "type": "ModelConfigRequest",
    },
    environment="string_example",
    feedback={
        "type": "string_example",
        "value": 3.14,
    },
    created_at="1970-01-01T00:00:00.00Z",
    error="string_example",
    duration=3.14,
    output_message={
        "role": "user",
    },
    prompt_tokens=1,
    output_tokens=1,
    prompt_cost=3.14,
    output_cost=3.14,
    provider_request={},
    provider_response={},
)

โš™๏ธ Parameters

project: str

Unique project name. If no project exists with this name, a new project will be created.

project_id: str

Unique ID of a project to associate to the log. Either this or project must be provided.

session_id: str

ID of the session to associate the datapoint.

session_reference_id: str

A unique string identifying the session to associate the datapoint to. Allows you to log multiple datapoints to a session (using an ID kept by your internal systems) by passing the same session_reference_id in subsequent log requests. Specify at most one of this or session_id.

parent_id: str

ID associated to the parent datapoint in a session.

parent_reference_id: str

A unique string identifying the previously-logged parent datapoint in a session. Allows you to log nested datapoints with your internal system IDs by passing the same reference ID as parent_id in a prior log request. Specify at most one of this or parent_id. Note that this cannot refer to a datapoint being logged in the same request.

inputs: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

The inputs passed to the prompt template.

source: str

Identifies where the model was called from.

metadata: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

Any additional metadata to record.

save: bool

Whether the request/response payloads will be stored on Humanloop.

source_datapoint_id: str

ID of the source datapoint if this is a log derived from a datapoint in a dataset.

reference_id: str

A unique string to reference the datapoint. Allows you to log nested datapoints with your internal system IDs by passing the same reference ID as parent_id in a subsequent log request.

trial_id: str

Unique ID of an experiment trial to associate to the log.

messages: List[ChatMessageWithToolCall]

The messages passed to the to provider chat endpoint.

output: str

Generated output from your model for the provided inputs. Can be None if logging an error, or if logging a parent datapoint with the intention to populate it later

judgment: Union[bool, Union[int, float]]
config_id: str

Unique ID of a config to associate to the log.

config: Union[ModelConfigRequest, ToolConfigRequest]

The model config used for this generation. Required unless config_id or trial_id is provided.

environment: str

The environment name used to create the log.

feedback: Union[Feedback, List[Feedback]]

Optional parameter to provide feedback with your logged datapoint.

created_at: datetime

User defined timestamp for when the log was created.

error: str

Error message if the log is an error.

duration: Union[int, float]

Duration of the logged event in seconds.

output_message: ChatMessageWithToolCall

The message returned by the provider.

prompt_tokens: int

Number of tokens in the prompt used to generate the output.

output_tokens: int

Number of tokens in the output generated by the model.

prompt_cost: Union[int, float]

Cost in dollars associated to the tokens in the prompt.

output_cost: Union[int, float]

Cost in dollars associated to the tokens in the output.

provider_request: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

Raw request sent to provider.

provider_response: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

Raw response received the provider.

โš™๏ธ Request Body

LogDatapointRequest

๐Ÿ”„ Return

LogsLogResponse

๐ŸŒ Endpoint

/logs post

๐Ÿ”™ Back to Table of Contents


humanloop.logs.update

Update a logged datapoint in your Humanloop project.

๐Ÿ› ๏ธ Usage

update_response = humanloop.logs.update(
    id="id_example",
    output="string_example",
    error="string_example",
    duration=3.14,
)

โš™๏ธ Parameters

id: str

String ID of logged datapoint to return. Starts with data_.

output: str

Generated output from your model for the provided inputs.

error: str

Error message if the log is an error.

duration: Union[int, float]

Duration of the logged event in seconds.

โš™๏ธ Request Body

UpdateLogRequest

๐Ÿ”„ Return

LogResponse

๐ŸŒ Endpoint

/logs/{id} patch

๐Ÿ”™ Back to Table of Contents


humanloop.logs.update_by_ref

Update a logged datapoint by its reference ID.

The reference_id query parameter must be provided, and refers to the reference_id of a previously-logged datapoint.

๐Ÿ› ๏ธ Usage

update_by_ref_response = humanloop.logs.update_by_ref(
    reference_id="reference_id_example",
    output="string_example",
    error="string_example",
    duration=3.14,
)

โš™๏ธ Parameters

reference_id: str

A unique string to reference the datapoint. Identifies the logged datapoint created with the same reference_id.

output: str

Generated output from your model for the provided inputs.

error: str

Error message if the log is an error.

duration: Union[int, float]

Duration of the logged event in seconds.

โš™๏ธ Request Body

UpdateLogRequest

๐Ÿ”„ Return

LogResponse

๐ŸŒ Endpoint

/logs patch

๐Ÿ”™ Back to Table of Contents


humanloop.model_configs.deserialize

Deserialize a model config from a .prompt file format.

๐Ÿ› ๏ธ Usage

deserialize_response = humanloop.model_configs.deserialize(
    config="string_example",
)

โš™๏ธ Parameters

config: str

โš™๏ธ Request Body

BodyModelConfigsDeserialize

๐Ÿ”„ Return

ModelConfigResponse

๐ŸŒ Endpoint

/model-configs/deserialize post

๐Ÿ”™ Back to Table of Contents


humanloop.model_configs.export

Export a model config to a .prompt file by ID.

๐Ÿ› ๏ธ Usage

export_response = humanloop.model_configs.export(
    id="id_example",
)

โš™๏ธ Parameters

id: str

String ID of the model config. Starts with config_.

๐ŸŒ Endpoint

/model-configs/{id}/export post

๐Ÿ”™ Back to Table of Contents


humanloop.model_configs.get

Get a specific model config by ID.

๐Ÿ› ๏ธ Usage

get_response = humanloop.model_configs.get(
    id="id_example",
)

โš™๏ธ Parameters

id: str

String ID of the model config. Starts with config_.

๐Ÿ”„ Return

ModelConfigResponse

๐ŸŒ Endpoint

/model-configs/{id} get

๐Ÿ”™ Back to Table of Contents


humanloop.model_configs.register

Register a model config to a project and optionally add it to an experiment.

If the project name provided does not exist, a new project will be created automatically.

If an experiment name is provided, the specified experiment must already exist. Otherwise, an error will be raised.

If the model config is the first to be associated to the project, it will be set as the active model config.

๐Ÿ› ๏ธ Usage

register_response = humanloop.model_configs.register(
    model="string_example",
    description="string_example",
    name="string_example",
    provider="openai",
    max_tokens=-1,
    temperature=1,
    top_p=1,
    stop="string_example",
    presence_penalty=0,
    frequency_penalty=0,
    other={},
    seed=1,
    response_format={
        "type": "json_object",
    },
    project="string_example",
    project_id="string_example",
    experiment="string_example",
    prompt_template="string_example",
    chat_template=[
        {
            "role": "user",
        }
    ],
    endpoint="complete",
    tools=[
        {
            "id": "id_example",
            "source": "organization",
        }
    ],
)

โš™๏ธ Parameters

model: str

The model instance used. E.g. text-davinci-002.

description: str

A description of the model config.

name: str

A friendly display name for the model config. If not provided, a name will be generated.

provider: ModelProviders

The company providing the underlying model service.

max_tokens: int

The maximum number of tokens to generate. Provide max_tokens=-1 to dynamically calculate the maximum number of tokens to generate given the length of the prompt

temperature: Union[int, float]

What sampling temperature to use when making a generation. Higher values means the model will be more creative.

top_p: Union[int, float]

An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass.

stop: Union[str, List[str]]

The string (or list of strings) after which the model will stop generating. The returned text will not contain the stop sequence.

presence_penalty: Union[int, float]

Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the generation so far.

frequency_penalty: Union[int, float]

Number between -2.0 and 2.0. Positive values penalize new tokens based on how frequently they appear in the generation so far.

other: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

Other parameter values to be passed to the provider call.

seed: int

If specified, model will make a best effort to sample deterministically, but it is not guaranteed.

response_format: ResponseFormat

The format of the response. Only type json_object is currently supported for chat.

project: str

Unique project name. If it does not exist, a new project will be created.

project_id: str

Unique project ID

experiment: str

If specified, the model config will be added to this experiment. Experiments are used for A/B testing and optimizing hyperparameters.

prompt_template: str

Prompt template that will take your specified inputs to form your final request to the provider model. NB: Input variables within the prompt template should be specified with syntax: {{INPUT_NAME}}.

chat_template: List[ChatMessageWithToolCall]

Messages prepended to the list of messages sent to the provider. These messages that will take your specified inputs to form your final request to the provider model. NB: Input variables within the prompt template should be specified with syntax: {{INPUT_NAME}}.

endpoint: ModelEndpoints

Which of the providers model endpoints to use. For example Complete or Edit.

tools: ProjectModelConfigRequestTools

โš™๏ธ Request Body

ProjectModelConfigRequest

๐Ÿ”„ Return

ProjectConfigResponse

๐ŸŒ Endpoint

/model-configs post

๐Ÿ”™ Back to Table of Contents


humanloop.model_configs.serialize

Serialize a model config to a .prompt file format.

๐Ÿ› ๏ธ Usage

serialize_response = humanloop.model_configs.serialize(
    body={
        "provider": "openai",
        "model": "model_example",
        "max_tokens": -1,
        "temperature": 1,
        "top_p": 1,
        "presence_penalty": 0,
        "frequency_penalty": 0,
        "endpoint": "complete",
    },
    description="string_example",
    name="string_example",
    provider="openai",
    model="string_example",
    max_tokens=-1,
    temperature=1,
    top_p=1,
    stop="string_example",
    presence_penalty=0,
    frequency_penalty=0,
    other={},
    seed=1,
    response_format={
        "type": "json_object",
    },
    endpoint="complete",
    chat_template=[
        {
            "role": "user",
        }
    ],
    tools=[
        {
            "id": "id_example",
            "source": "organization",
        }
    ],
    prompt_template="{{question}}",
)

โš™๏ธ Parameters

description: str

A description of the model config.

name: str

A friendly display name for the model config. If not provided, a name will be generated.

provider: ModelProviders

The company providing the underlying model service.

model: str

The model instance used. E.g. text-davinci-002.

max_tokens: int

The maximum number of tokens to generate. Provide max_tokens=-1 to dynamically calculate the maximum number of tokens to generate given the length of the prompt

temperature: Union[int, float]

What sampling temperature to use when making a generation. Higher values means the model will be more creative.

top_p: Union[int, float]

An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass.

stop: Union[str, List[str]]

The string (or list of strings) after which the model will stop generating. The returned text will not contain the stop sequence.

presence_penalty: Union[int, float]

Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the generation so far.

frequency_penalty: Union[int, float]

Number between -2.0 and 2.0. Positive values penalize new tokens based on how frequently they appear in the generation so far.

other: Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]

Other parameter values to be passed to the provider call.

seed: int

If specified, model will make a best effort to sample deterministically, but it is not guaranteed.

response_format: ResponseFormat

The format of the response. Only type json_object is currently supported for chat.

endpoint: ModelEndpoints

The provider model endpoint used.

chat_template: List[ChatMessageWithToolCall]

Messages prepended to the list of messages sent to the provider. These messages that will take your specified inputs to form your final request to the provider model. Input variables within the template should be specified with syntax: {{INPUT_NAME}}.

tools: ModelConfigChatRequestTools
prompt_template: str

Prompt template that will take your specified inputs to form your final request to the model. Input variables within the prompt template should be specified with syntax: {{INPUT_NAME}}.

โš™๏ธ Request Body

ModelConfigsSerializeRequest

๐ŸŒ Endpoint

/model-configs/serialize post

๐Ÿ”™ Back to Table of Contents


humanloop.projects.create

Create a new project.

๐Ÿ› ๏ธ Usage

create_response = humanloop.projects.create(
    name="string_example",
    feedback_types=[
        {
            "type": "type_example",
            "_class": "select",
        }
    ],
    directory_id="string_example",
)

โš™๏ธ Parameters

name: str

Unique project name.

feedback_types: List[FeedbackTypeRequest]

Feedback types to be created.

directory_id: str

ID of directory to assign project to. Starts with dir_. If not provided, the project will be created in the root directory.

โš™๏ธ Request Body

CreateProjectRequest

๐Ÿ”„ Return

ProjectResponse

๐ŸŒ Endpoint

/projects post

๐Ÿ”™ Back to Table of Contents


humanloop.projects.create_feedback_type

Create Feedback Type

๐Ÿ› ๏ธ Usage

create_feedback_type_response = humanloop.projects.create_feedback_type(
    type="string_example",
    id="id_example",
    values=[
        {
            "value": "value_example",
            "sentiment": "positive",
        }
    ],
    _class="select",
)

โš™๏ธ Parameters

type: str

The type of feedback to update.

id: str

String ID of project. Starts with pr_.

values: List[FeedbackLabelRequest]

The feedback values to be available. This field should only be populated when updating a 'select' or 'multi_select' feedback class.

_class: FeedbackClass

The data type associated to this feedback type; whether it is a 'text'/'select'/'multi_select'. This is optional when updating the default feedback types (i.e. when type is 'rating', 'action' or 'issue').

โš™๏ธ Request Body

FeedbackTypeRequest

๐Ÿ”„ Return

FeedbackTypeModel

๐ŸŒ Endpoint

/projects/{id}/feedback-types post

๐Ÿ”™ Back to Table of Contents


humanloop.projects.deactivate_config

Remove the project's active config, if set.

This has no effect if the project does not have an active model config set.

๐Ÿ› ๏ธ Usage

deactivate_config_response = humanloop.projects.deactivate_config(
    id="id_example",
    environment="string_example",
)

โš™๏ธ Parameters

id: str

String ID of project. Starts with pr_.

environment: str

Name for the environment. E.g. 'production'. If not provided, will delete the active config for the default environment.

๐Ÿ”„ Return

ProjectResponse

๐ŸŒ Endpoint

/projects/{id}/active-config delete

๐Ÿ”™ Back to Table of Contents


humanloop.projects.deactivate_experiment

Remove the project's active experiment, if set.

This has no effect if the project does not have an active experiment set.

๐Ÿ› ๏ธ Usage

deactivate_experiment_response = humanloop.projects.deactivate_experiment(
    id="id_example",
    environment="string_example",
)

โš™๏ธ Parameters

id: str

String ID of project. Starts with pr_.

environment: str

Name for the environment. E.g. 'producton'. If not provided, will return the experiment for the default environment.

๐Ÿ”„ Return

ProjectResponse

๐ŸŒ Endpoint

/projects/{id}/active-experiment delete

๐Ÿ”™ Back to Table of Contents


humanloop.projects.delete

Delete a specific file.

๐Ÿ› ๏ธ Usage

humanloop.projects.delete(
    id="id_example",
)

โš™๏ธ Parameters

id: str

String ID of project. Starts with pr_.

๐ŸŒ Endpoint

/projects/{id} delete

๐Ÿ”™ Back to Table of Contents


humanloop.projects.delete_deployed_config

Remove the version deployed to environment.

This has no effect if the project does not have an active version set.

๐Ÿ› ๏ธ Usage

delete_deployed_config_response = humanloop.projects.delete_deployed_config(
    project_id="project_id_example",
    environment_id="environment_id_example",
)

โš™๏ธ Parameters

project_id: str
environment_id: str

๐ŸŒ Endpoint

/projects/{project_id}/deployed-config/{environment_id} delete

๐Ÿ”™ Back to Table of Contents


humanloop.projects.deploy_config

Deploy a model config to an environment.

If the environment already has a model config deployed, it will be replaced.

๐Ÿ› ๏ธ Usage

deploy_config_response = humanloop.projects.deploy_config(
    project_id="project_id_example",
    config_id="string_example",
    experiment_id="string_example",
    environments=[
        {
            "id": "id_example",
        }
    ],
)

โš™๏ธ Parameters

project_id: str
config_id: str

Model config unique identifier generated by Humanloop.

experiment_id: str

String ID of experiment. Starts with exp_.

environments: List[EnvironmentRequest]

List of environments to associate with the model config.

โš™๏ธ Request Body

EnvironmentProjectConfigRequest

๐Ÿ”„ Return

ProjectsDeployConfigToEnvironmentsResponse

๐ŸŒ Endpoint

/projects/{project_id}/deploy-config patch

๐Ÿ”™ Back to Table of Contents


humanloop.projects.export

Export all logged datapoints associated to your project.

Results are paginated and sorts the datapoints based on created_at in descending order.

๐Ÿ› ๏ธ Usage

export_response = humanloop.projects.export(
    id="id_example",
    page=0,
    size=10,
)

โš™๏ธ Parameters

id: str

String ID of project. Starts with pr_.

page: int

Page offset for pagination.

size: int

Page size for pagination. Number of logs to export.

๐Ÿ”„ Return

PaginatedDataLogResponse

๐ŸŒ Endpoint

/projects/{id}/export post

๐Ÿ”™ Back to Table of Contents


humanloop.projects.get

Get a specific project.

๐Ÿ› ๏ธ Usage

get_response = humanloop.projects.get(
    id="id_example",
)

โš™๏ธ Parameters

id: str

String ID of project. Starts with pr_.

๐Ÿ”„ Return

ProjectResponse

๐ŸŒ Endpoint

/projects/{id} get

๐Ÿ”™ Back to Table of Contents


humanloop.projects.get_active_config

Retrieves a config to use to execute your model.

A config will be selected based on the project's active config/experiment settings.

๐Ÿ› ๏ธ Usage

get_active_config_response = humanloop.projects.get_active_config(
    id="id_example",
    environment="string_example",
)

โš™๏ธ Parameters

id: str

String ID of project. Starts with pr_.

environment: str

Name for the environment. E.g. 'producton'. If not provided, will return the active config for the default environment.

๐Ÿ”„ Return

GetModelConfigResponse

๐ŸŒ Endpoint

/projects/{id}/active-config get

๐Ÿ”™ Back to Table of Contents


humanloop.projects.list

Get a paginated list of files.

๐Ÿ› ๏ธ Usage

list_response = humanloop.projects.list(
    page=0,
    size=10,
    filter="string_example",
    user_filter="string_example",
    sort_by="created_at",
    order="asc",
)

โš™๏ธ Parameters

page: int

Page offset for pagination.

size: int

Page size for pagination. Number of projects to fetch.

filter: str

Case-insensitive filter for project name.

user_filter: str

Case-insensitive filter for users in the project. This filter matches against both email address and name of users.

sort_by: ProjectSortBy

Field to sort projects by

order: SortOrder

Direction to sort by.

๐Ÿ”„ Return

PaginatedDataProjectResponse

๐ŸŒ Endpoint

/projects get

๐Ÿ”™ Back to Table of Contents


humanloop.projects.list_configs

Get an array of versions associated to your file.

๐Ÿ› ๏ธ Usage

list_configs_response = humanloop.projects.list_configs(
    id="id_example",
    evaluation_aggregates=True,
)

โš™๏ธ Parameters

id: str

String ID of project. Starts with pr_.

evaluation_aggregates: bool

๐Ÿ”„ Return

ProjectsGetConfigsResponse

๐ŸŒ Endpoint

/projects/{id}/configs get

๐Ÿ”™ Back to Table of Contents


humanloop.projects.list_deployed_configs

Get an array of environments with the deployed configs associated to your project.

๐Ÿ› ๏ธ Usage

list_deployed_configs_response = humanloop.projects.list_deployed_configs(
    id="id_example",
)

โš™๏ธ Parameters

id: str

String ID of project. Starts with pr_.

๐Ÿ”„ Return

ProjectsGetDeployedConfigsResponse

๐ŸŒ Endpoint

/projects/{id}/deployed-configs get

๐Ÿ”™ Back to Table of Contents


humanloop.projects.update

Update a specific project.

Set the project's active model config/experiment by passing either active_experiment_id or active_model_config_id. These will be set to the Default environment unless a list of environments are also passed in specifically detailing which environments to assign the active config or experiment.

Set the feedback labels to be treated as positive user feedback used in calculating top-level project metrics by passing a list of labels in positive_labels.

๐Ÿ› ๏ธ Usage

update_response = humanloop.projects.update(
    id="id_example",
    name="string_example",
    active_experiment_id="string_example",
    active_config_id="string_example",
    positive_labels=[
        {
            "type": "type_example",
            "value": "value_example",
        }
    ],
    directory_id="string_example",
)

โš™๏ธ Parameters

id: str

String ID of project. Starts with pr_.

name: str

The new unique project name. Caution, if you are using the project name as the unique identifier in your API calls, changing the name will break the calls.

active_experiment_id: str

ID for an experiment to set as the project's active deployment. Starts with 'exp_'. At most one of 'active_experiment_id' and 'active_model_config_id' can be set.

active_config_id: str

ID for a config to set as the project's active deployment. Starts with 'config_'. At most one of 'active_experiment_id' and 'active_config_id' can be set.

positive_labels: List[PositiveLabel]

The full list of labels to treat as positive user feedback.

directory_id: str

ID of directory to assign project to. Starts with dir_.

โš™๏ธ Request Body

UpdateProjectRequest

๐Ÿ”„ Return

ProjectResponse

๐ŸŒ Endpoint

/projects/{id} patch

๐Ÿ”™ Back to Table of Contents


humanloop.projects.update_feedback_types

Update feedback types.

Allows enabling the available feedback types and setting status of feedback types/categorical values.

This behaves like an upsert; any feedback categorical values that do not already exist in the project will be created.

๐Ÿ› ๏ธ Usage

update_feedback_types_response = humanloop.projects.update_feedback_types(
    body=[
        {
            "type": "type_example",
            "_class": "select",
        }
    ],
    id="id_example",
)

โš™๏ธ Parameters

id: str

String ID of project. Starts with pr_.

requestBody: ProjectsUpdateFeedbackTypesRequest

๐Ÿ”„ Return

FeedbackTypes

๐ŸŒ Endpoint

/projects/{id}/feedback-types patch

๐Ÿ”™ Back to Table of Contents


humanloop.sessions.create

Create a new session.

Returns a session ID that can be used to log datapoints to the session.

๐Ÿ› ๏ธ Usage

create_response = humanloop.sessions.create()

๐Ÿ”„ Return

CreateSessionResponse

๐ŸŒ Endpoint

/sessions post

๐Ÿ”™ Back to Table of Contents


humanloop.sessions.get

Get a session by ID.

๐Ÿ› ๏ธ Usage

get_response = humanloop.sessions.get(
    id="id_example",
)

โš™๏ธ Parameters

id: str

String ID of session to return. Starts with sesh_.

๐Ÿ”„ Return

SessionResponse

๐ŸŒ Endpoint

/sessions/{id} get

๐Ÿ”™ Back to Table of Contents


humanloop.sessions.list

Get a page of sessions.

๐Ÿ› ๏ธ Usage

list_response = humanloop.sessions.list(
    project_id="project_id_example",
    page=1,
    size=10,
)

โš™๏ธ Parameters

project_id: str

String ID of project to return sessions for. Sessions that contain any datapoints associated to this project will be returned. Starts with pr_.

page: int

Page to fetch. Starts from 1.

size: int

Number of sessions to retrieve.

๐Ÿ”„ Return

PaginatedDataSessionResponse

๐ŸŒ Endpoint

/sessions get

๐Ÿ”™ Back to Table of Contents


Author

This Python package is automatically generated by Konfig

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

humanloop-0.7.30.tar.gz (332.0 kB view hashes)

Uploaded Source

Built Distribution

humanloop-0.7.30-py3-none-any.whl (1.4 MB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page