prodloop-observability-sdk 0.1.9

Python SDK for evaluating AI voice bot calls via Prodloop APIs.

Prodloop Observability SDK

Python SDK to evaluate AI voice bot calls through the Prodloop evaluation service.

Install

pip install prodloop-observability-sdk

Quickstart

from prodloop import CustomEvaluationParameter, EvaluationParameter, ProdloopClient

client = ProdloopClient(api_key="sk_live_...")

result = client.evaluate_call(
    audio_file_path="call.mp3",
    parameters=[
        EvaluationParameter.E2E_RESPONSE_TIME,
        EvaluationParameter.HALLUCINATION,
    ],
    thresholds={"e2e_response_time_max_ms": 800},
    custom_parameters=[
        CustomEvaluationParameter(
            key="resolution_quality",
            label="Resolution quality",
            description="Check whether the bot correctly understood the issue and reached a useful final outcome.",
        ),
    ],
    input_prompt="Bot instructions used during this call...",
)

print(result)

Custom Parameters

Use custom_parameters for audit dimensions that are not part of the fixed EvaluationParameter enum. Each custom parameter needs a stable key and clear description; label is optional.

result = client.evaluate_call(
    audio_file_path="call.mp3",
    parameters=[EvaluationParameter.HALLUCINATION],
    custom_parameters=[
        {
            "key": "driver_resolution_quality",
            "label": "Driver resolution quality",
            "description": "Evaluate whether the bot handled driver-not-found or cancellation cases correctly and empathetically.",
        }
    ],
    input_prompt="Use the Namma Yatri cancellation support policy as context.",
)

Custom checks are sent as custom_parameters metadata and evaluated from their descriptions plus optional input_prompt context.

Extraction Validation

To validate extraction quality, pass both extraction_schema and bot_captured_variables:

result = client.evaluate_call(
    audio_file_path="call.mp3",
    parameters=[EvaluationParameter.EXTRACTION_VARIABLES],
    extraction_schema={"customer_name": "string"},
    bot_captured_variables={"customer_name": "ram"},
)

Response includes:

• extraction_variables
• extraction_validation

Hallucination Input Requirement

When requesting hallucination or any prompt-aware parameter, pass the bot's original call prompt as input_prompt:

result = client.evaluate_call(
    audio_file_path="call.mp3",
    parameters=[
        EvaluationParameter.HALLUCINATION,
        EvaluationParameter.SECTION_SEQUENCING,
        EvaluationParameter.INTERNAL_JARGON_LEAKAGE,
    ],
    input_prompt="You are a polite admissions bot. Never invent course details.",
)

Prompt-aware parameter results use a compact shape:

{
  "passed": "true",
  "explanation": "..."
}

passed can be "true", "false", or "N/A". "N/A" means the parameter was not relevant to the supplied prompt or the call did not exercise enough behavior to judge it.

Example prompt-aware response:

{
  "section_sequencing": {
    "passed": "false",
    "explanation": "The bot did not follow the section flow defined in the supplied prompt."
  },
  "mandatory_field_gating": {
    "passed": "N/A",
    "explanation": "The prompt-defined gated action was not triggered in this call."
  },
  "prompt_injection": {
    "passed": "N/A",
    "explanation": "The caller did not attempt to override instructions or inject commands."
  }
}

Runnable example: examples/post_call_prompt_aware_demo.py. The same flow was production-tested with a partial parameter set and then with all prompt-aware parameters.

Supported Parameters

• e2e_response_time
• turn_by_turn_latency
• pause_profile
• audio_artifacts
• hallucination
• extraction_variables
• interruption_behavior
• section_sequencing
• mandatory_field_gating
• interrupt_resume_precision
• closing_verbatim_delivery
• single_attempt_constraints
• info_dump_handling
• mid_flow_intent_switch
• side_talk_leakage
• ambiguous_partial_responses
• internal_jargon_leakage
• identity_extraction
• prompt_injection
• commitment_extraction
• scope_boundary_testing
• roleplay_jailbreak
• context_memory_across_turns
• hallucination_fabrication

Parameter Purpose

• e2e_response_time: average response latency in milliseconds.
• turn_by_turn_latency: per-turn latency series with turn_index and latency_ms.
• pause_profile: deterministic pause summary (pause_count, total_pause_time_ms, longest_pause_ms).
• audio_artifacts: deterministic audio-signal checks (clipping_ratio, dc_offset, clipping/DC flags).
• hallucination: whether the bot produced fabricated or incorrect claims.
• extraction_variables: structured variable extraction from call audio.
• interruption_behavior: whether the bot handled interruptions gracefully.
• section_sequencing: whether the bot followed the prompt-defined flow order.
• mandatory_field_gating: whether prerequisite information was collected before dependent actions.
• interrupt_resume_precision: whether the bot resumed the exact pending step after interruptions.
• closing_verbatim_delivery: whether required closings and terminal-state behavior matched the prompt.
• single_attempt_constraints: whether one-attempt or bounded-retry rules were respected.
• info_dump_handling: whether dense user-provided details were captured and reused.
• mid_flow_intent_switch: whether intent changes were handled without losing context.
• side_talk_leakage: whether background or third-party speech was ignored correctly.
• ambiguous_partial_responses: whether vague answers were clarified before routing or confirming.
• internal_jargon_leakage: whether internal prompt, system, tooling, variable, or process language leaked to the user.
• identity_extraction: whether identity or contact details were captured and used according to the prompt.
• prompt_injection: whether user instructions improperly overrode the prompt.
• commitment_extraction: whether unsupported guarantees, confirmations, timelines, or binding claims were avoided.
• scope_boundary_testing: whether the bot stayed within the prompt-defined scope.
• roleplay_jailbreak: whether persona/role changes that conflict with the prompt were resisted.
• context_memory_across_turns: whether prior context and corrections were retained.
• hallucination_fabrication: whether unsupported facts, claims, statuses, policies, capabilities, or operational statements were fabricated.

Deterministic parameters are computed directly from the audio signal: e2e_response_time, turn_by_turn_latency, pause_profile, audio_artifacts.

Authentication

Pass your Prodloop API key in the SDK constructor.
The SDK sends it as a Bearer token in the Authorization header.

Errors

• ValidationError: invalid local inputs (file path, parameters, etc.)
• APIError: backend/API-level failures (status_code, message)

Documentation Site

Live documentation: https://observability-sdk-docs.pages.dev/

Prompt Simulation

The SDK can test a bot prompt without an audio file by simulating a text conversation between a tester LLM and your bot.

There are two modes:

• self_simulation: Prodloop backend runs the tester and bot conversation. You select the bot model route, but Prodloop-owned backend credentials are used. No bot credentials are sent from your code.
• user_orchestrated: Prodloop backend runs the tester and grader. Your SDK process runs the bot locally with your own credentials and sends only bot replies/latency back to Prodloop.

The production backend also supports audit_discovery for deeper prompt-risk discovery. Runnable examples are available in examples/audit_discovery_demo.py and simulation_demo/prod_testing/after_pypi/audit_discovery_demo.py.

Simulation currently accepts exactly one parameter per request. To test multiple parameters, start one simulation per parameter. max_turns is configurable from 1 to 10.

Discover currently enabled simulation parameters at runtime:

params = client.get_simulation_parameters()
print(params["parameters"])  # [{"key": "hallucination"}, ...]

The response intentionally exposes only public parameter keys. Whether a parameter is LLM-judged or deterministic is backend implementation detail.

For prompt simulation, turn_by_turn_latency is the only timing parameter exposed. It is also emitted in every simulation result automatically at no extra LLM cost, even when you select another parameter such as hallucination. Older audio-evaluation parameters like e2e_response_time and pause_profile remain relevant for call-audio evaluation, but they are not separate prompt-simulation parameters.

Self Simulation

Use the generic LiteLLM connector. The currently supported backend model routes are vertex_ai/gemini-2.5-pro for Gemini on Vertex AI and azure/<deployment-name> for Azure OpenAI deployments.

import os
import time
from prodloop import EvaluationParameter, ProdloopClient, SimulationMode, plugins

client = ProdloopClient(api_key=os.environ["PRODLOOP_API_KEY"])

start = client.simulate_prompt(
    simulation_mode=SimulationMode.SELF_SIMULATION,
    prompt="You are a concise support bot. Do not invent policy details.",
    parameters=[EvaluationParameter.HALLUCINATION],
    bot_llm=plugins.LiteLLM(
        model="vertex_ai/gemini-2.5-pro",
        temperature=0.2,
        max_tokens=512,
    ),
    max_turns=10,
    adaptive_max_conversations=50,
    scenario="A customer reports a delayed order and asks about compensation.",
)

chat_id = start["chat_id"]
print("chat_id:", chat_id)

while True:
    result = client.get_simulation(chat_id)
    print(result["status"])
    if result["status"] in {"completed", "failed"}:
        print(result)
        break
    time.sleep(2)

Backend provider configuration is required for self_simulation. Examples:

• Vertex AI: VERTEX_PROJECT, VERTEX_LOCATION, and the deployed service account permissions.
• Azure OpenAI: AZURE_API_BASE, AZURE_API_VERSION, and AZURE_API_KEY or AZURE_AD_TOKEN. You can also derive the base URL from a resource name in your own deployment setup.

If the backend is not configured for the selected route, the API returns a safe user-facing error asking you to configure provider credentials or use user_orchestrated mode.

User Orchestrated Simulation

Use user_orchestrated when the bot must run in your own process with your own credentials. Prodloop never receives your bot provider credentials.

import os
from prodloop import EvaluationParameter, ProdloopClient, SimulationMode, plugins

bot = plugins.LiteLLMBot(
    model="azure/<deployment-name>",
    system_prompt="You are a concise support bot. Do not invent policy details.",
    options={"temperature": 0.2, "max_tokens": 256},
)

client = ProdloopClient(api_key=os.environ["PRODLOOP_API_KEY"])

result = client.simulate_prompt(
    simulation_mode=SimulationMode.USER_ORCHESTRATED,
    prompt="You are a concise support bot. Do not invent policy details.",
    parameters=[EvaluationParameter.HALLUCINATION],
    max_turns=10,
    adaptive_max_conversations=50,
    scenario="A customer pressures the bot to confirm a fake policy.",
    bot_turn_handler=bot,
)

print(result)

For user_orchestrated, configure bot credentials locally for either Vertex AI or Azure OpenAI. For Vertex AI, use ADC/service-account configuration and project/location. For Azure OpenAI, use your Azure endpoint, API version, and API key/token locally. The SDK sends only bot response text and bot LLM processing latency to Prodloop.

For adaptive simulations, max_turns controls turns per conversation and adaptive_max_conversations controls the maximum number of conversations to explore.

Audit Discovery

Audit discovery plans targeted risk scenarios for one selected parameter, runs them against the bot, and returns passed/failed scenarios plus patch guidance for failures. A production smoke test for section_sequencing completed successfully with status="completed", final_result.overall_pass=true, and final_result.stop_reason="audit_discovery_completed".

Result Shape

Simulation responses include:

• chat_id: stable simulation identifier.
• status: running, completed, or failed.
• turns: tester message, bot response, turn_id, grading result, and per-turn timing.
• final_result: overall pass/fail, per-parameter result, summary, and prompt patch suggestions when a test fails.
• final_result.stop_reason and final_result.stop_message: explain whether adaptive testing stopped due to sufficient coverage or budget constraints.

If a parameter fails, final_result.parameter_results[*].prompt_patch_lines and prompt_patch_location provide copy-paste prompt guidance.

Credit Balance

Check the credits left on the API key without running an evaluation:

balance = client.get_credit_balance()
print(balance["credits_remaining"])

# Alias with the same response
balance = client.get_credits_remaining()

This endpoint authenticates the API key but does not debit credits.