ptm-client

Lightweight Python client for the Prompt Test Manager (PTM) API. Single runtime dependency: requests.

Install

pip install ptm-client

Quick start

from ptm_client import PTMClient

client = PTMClient(base_url="https://ptm.example.com", token="your-api-token")

# List prompts, filtered by tag
prompts = client.list_prompts(tag="my_team")

# Fetch a prompt and its test cases
detail = client.get_prompt("my_team.summarizer")
tests = client.get_prompt_tests("my_team.summarizer")

# Run an eval against the library prompt
run = client.run_eval(
    prompt_ids=["my_team.summarizer"],
    provider_ids=["openai_gpt55"],
)

# Or run a one-off manual eval
run = client.run_manual_eval(
    {
        "prompt_text": "Summarize: {{text}}",
        "tests": [{"description": "smoke", "vars": {"text": "Hello, world."}}],
        "provider_profiles": ["openai_gpt55"],
    }
)

# Block until complete, then fetch a report
result = client.wait_for_run(run["run_key"], timeout=120)
html = client.run_report(run["run_key"])
json_report = client.run_report(run["run_key"], format="json")

Constructor

PTMClient(base_url, token, timeout=30, *, verify_ssl=None)

token is a PTM personal access token or service account token. timeout is the HTTP request timeout in seconds.

verify_ssl controls TLS certificate verification. When None (the default), the value is read from the PTM_SSL_VERIFY environment variable, which defaults to true. Set PTM_SSL_VERIFY=false (or pass verify_ssl=False) to allow self-signed or otherwise invalid certificates - intended for local dev against a homelab PTM instance, never for production. The matching urllib3.InsecureRequestWarning is silenced when verification is disabled.

Public methods

Prompts

• list_prompts(tag=None, team=None, service=None, source=None, search=None, group=None)
• get_prompt(prompt_id)
• get_prompt_tests(prompt_id)
• list_prompt_versions(prompt_id) (v0.3.0)
• get_prompt_version(prompt_id, version_number) (v0.3.0)

Providers

• list_providers()

Evaluations

• run_eval(prompt_ids, provider_ids, **kwargs)
• run_manual_eval(payload)
• run_prompt_eval(prompt_id, provider_ids, *, inject_vars=None, extra_tests=None, visibility_scope="org_visible", label=None)

Runs

• list_runs(limit=50, terminal_only=False, mine_only=False) (v0.3.0)
• get_run(run_key)
• wait_for_run(run_key, timeout=300, poll_interval=5)
• wait_for_security_scan(scan_id, *, timeout=300, poll_interval=3)
• run_report(run_key, format="html") - html / json / markdown / csv

Optimization (v0.3.0)

• submit_optimization(prompt_id, provider_profiles=None, judge_profile=None, max_cycles=10, target_score=90.0, min_improvement=2.0, max_cost_usd=20.0, comparison_strategy=None, visibility_scope=None, *, stability_samples=None, validation_samples=None, flakiness_threshold=None, min_consistent_improvement=None, variance_aware_mutator=None, variance_signal=None, enforce_target_score=None, chained_baseline_mode=None) (variance-aware fields added in v0.6.0; chained_baseline_mode in v0.13.0; omit any to inherit server admin defaults)
• optimize_prompt(...) - deprecated alias for submit_optimization; emits DeprecationWarning; removed in v1.0.0
• get_optimization_status(prompt_id)
• get_optimization_history(prompt_id)
• get_optimization_detail(optimization_id) (v0.3.0)
• cancel_optimization(optimization_id)
• wait_for_optimization(prompt_id, *, timeout=600, poll_interval=10)

Skills (v0.10.0)

• list_skills(namespace=None, search=None, tags=None, include_deprecated=False, page=1, page_size=50, held=None, source=None, kind=None) - held=True lists only skills on a security hold (with disabled_reason, disabled_at, disabled_by); source (a skill source slug) and kind (owned / linked) (v0.24.0) narrow to linked skills
• get_skill(skill_id_or_qname)
• get_skill_by_name(qualified_name)
• get_skills_by_prompt(prompt_id)
• publish_skill(prompt_id, payload, *, skill_name=None)
• update_skill(skill_id, payload)
• delete_skill(skill_id)
• install_skill_for_claude_code(skill_id, *, install_name=None) - writes bundle to ~/.claude/skills/{slug}/
• invoke_skill(skill_id, inputs, *, provider_profile=None, amend=False) (v0.10.0; amend added v0.14.0; DEPRECATED in v0.15.0 - emits DeprecationWarning and routes to invoke_skill_eval_ptm)
• invoke_skill_by_name(qualified_name, inputs, *, provider_profile=None, amend=False) (v0.14.0; in v0.15.0 routes to invoke_skill_eval_ptm)
• invoke_prompt(prompt_id, inputs, *, provider_profile=None, amend=False) (v0.10.0; amend added v0.14.0; DEPRECATED in v0.15.0 - routes to invoke_prompt_eval_ptm)

Skill sources (v0.24.0, server 2.7.0)

Third-party Agent Skills repositories and Claude plugin marketplaces an admin or IT admin approved. Their skills are linked skills, named <source>:<name>, read-only, and installed into a folder named after their install_name (the upstream SKILL.md name), so an install is a spec-conformant Agent Skills folder for Claude Code, Codex or Hermes.

• list_skill_sources(trust_state=None, kind=None) - {"items": [...]}; policy and sync health only for manage_skill_sources holders.
• request_skill_source(url, *, reason, provider=None, audience="org", audience_group_id=None) - request an https repository (provider is required for a self-hosted host); nothing syncs until it is approved.
• list_skill_source_entries(source_id, status=None, page=1, page_size=50) - every skill folder found upstream with its status (quarantined, available, blocked, unsupported, ...) and reason.
• With skill sources disabled on the deployment these raise PTMNotFoundError.

Security scanning (v0.22.0)

• run_security_scan(target_type, target_id, *, deep=False, version_number=None, judge_profile=None, wait=False, timeout=300) - target_type is "skill" (UUID) or "prompt" (prompt_id). Every scan is queued on the worker and returns the pending row; wait=True blocks for the final row (completed, or status: "error", never a fabricated safe verdict) and raises PTMTimeoutError after timeout seconds. deep=True runs an LLM-assisted scan. judge_profile (deep only) picks a judge-capable provider profile instead of the org default; the CLI flag is --judge. Each finding carries evidence, explanation and remediation when the scanner provides them.
• get_security_scan(scan_id) (v0.23.0) - one scan row by id, for polling a queued scan; 404 when the caller cannot see its target.
• list_security_scans(target_type, target_id, *, limit=10, offset=0, version_number=None) - history, newest first; limit is 1-100.
• The skill target_id is the UUID; the CLI also accepts a qualified name or bare public slug and looks it up first. With scanning disabled on the deployment both methods raise PTMNotFoundError ("Security scanning is not enabled"). Findings are attacker-authored text: display them, never execute them.
• A skill on security hold raises PTMLockedError (HTTP 423) from install, download, contents and invoke, and from the backing prompt's get_prompt_contents, get_prompt_version and invoke methods for callers who cannot edit the prompt and lack manage_skill_catalog. exc.reason is the manager's full recorded reason (the message only embeds a truncated preview of the body).
• A block-mode security gate raises PTMValidationError whose security_block holds the structured detail (verdict, risk_score, findings_total, findings); publishing the prompt of a held skill raises PTMConflictError whose security_hold holds the detail. Both messages name the security cause instead of the generic shape / retry hint.

Inline-vs-eval execution surfaces (v0.15.0)

PTM 1.16.x exposes three execution modes per kind (skill + prompt). Use the helper that matches the user's intent.

• invoke_skill_eval_ptm(skill_id, inputs, provider_profile, *, judge_profile=None, timeout_seconds=60, amend=False) - blocking server-side eval; returns scored output and retains a Eval: <qualified_name> row in run history.
• invoke_prompt_eval_ptm(prompt_id, inputs, provider_profile, ...) - prompt twin.
• invoke_skill_with_bundle(skill_id, inputs, provider_profile, ...) - returns the skill bundle for local execution AND enqueues a parallel server-side scored eval. Does not block on the eval. Response carries the bundle plus run_key.
• invoke_skill_with_bundle_by_name(qualified_name, ...) - resolves the qualified name and forwards.
• invoke_prompt_with_bundle(prompt_id, inputs, provider_profile, ...) - prompt twin.
• record_skill_client_inline_run(skill_id, *, provider_label, duration_ms, input_keys, success) - 204 telemetry for client-side inline executions. All four kwargs are REQUIRED server-side; passing any one omitted raises HTTP 422. The endpoint sees only the names of input keys, never their values.
• record_prompt_client_inline_run(prompt_id, *, provider_label, duration_ms, input_keys, success) - prompt twin.

Contents + usage metrics (v0.14.0)

• get_skill_contents(skill_id_or_qname) - fetch bundle in memory as {prompt_md, manifest, files}; no disk write, no LLM call
• get_prompt_contents(prompt_id, *, version=None) - fetch prompt contract as {prompt_text, version, tests, deepeval_metrics, tool_definitions}
• get_skill_usage_metrics(skill_id) - {lifetime, last_30d, last_7d, last_request_at} aggregated from fetch-event audit rows; returns null shape when org setting inline_usage_metrics_enabled=false
• get_prompt_usage_metrics(prompt_id) - same shape for prompt fetch events

Groups (v0.13.0)

• list_groups()
• create_prompt(payload)

Test-case shapes

PTM evaluates with three optional scoring layers. Use any combination.

Promptfoo assertions (deterministic)

Go in the assert array inside each test case:

{
    "description": "mention the topic with enough length",
    "vars": {"transcript": "..."},
    "assert": [
        {"type": "icontains", "value": "API migration"},
        {"type": "javascript", "value": "output.length >= 100"},
    ],
}

DeepEval metrics (semantic, judge-LLM)

Go in additional_metrics at the payload root:

{
    "additional_metrics": [
        {"name": "relevance", "criteria": "Output addresses the input topic.", "threshold": 0.7},
    ],
    "judge_profile": "openai_gpt56_terra",
}

KPI configs (custom weighted expressions)

Go in additional_kpis at the payload root:

{
    "additional_kpis": [
        {
            "name": "cost_ok",
            "description": "Under $0.05",
            "expression": "1 if cost < 0.05 else 0",
            "weight": 1.0,
        },
    ],
}

Inline examples

run_manual_eval - full control

run = client.run_manual_eval(
    {
        "label": "my_custom_eval",
        "prompt_text": "Summarize: {{text}}",
        "tests": [{"description": "short text", "vars": {"text": "The quick brown fox."}}],
        "provider_profiles": ["openai_gpt55"],
        "cost_threshold": 1.0,
        "latency_threshold_ms": 30000,
    }
)

run_prompt_eval - fetch from PTM + inject live data

run = client.run_prompt_eval(
    prompt_id="my_team.summarizer",
    provider_ids=["openai_gpt55"],
    inject_vars={"transcript": real_transcript, "meeting_title": "Weekly 1:1"},
)
result = client.wait_for_run(run["run_key"], timeout=120)

Error handling

from ptm_client import PTMClient, PTMError, PTMTimeoutError

try:
    result = client.wait_for_run(run_key, timeout=60)
except PTMTimeoutError:
    print("Run did not complete in time")
except PTMError as e:
    print(f"PTM API error ({e.status_code}): {e}")

PTMError wraps all HTTP errors, ConnectionError, and requests.Timeout. Check e.status_code (0 for connection/timeout failures). Typed subclasses: PTMPermissionError (403), PTMNotFoundError (404), PTMConflictError (409; .security_hold on a held-skill republish), PTMValidationError (422; .security_block on a block-mode security gate), PTMLockedError (423, security hold; .reason).

CLI

ptm (or python -m ptm_client) reads PTM_API_BASE_URL and PTM_API_TOKEN from the environment.

For both scan commands a skill <id> may be a UUID, a qualified name or a bare public slug. An invalid combination (--judge without --deep, --version on a skill) prints Invalid arguments: ... and exits 2 without sending the scan request.

Compatibility

• Python 3.12+
• PTM backend >= 1.9.0 required. get_skill_contents, get_prompt_contents, and usage-metrics methods require PTM >= 1.16.0. Skill invoke methods require PTM >= 1.13.1. Older backends work for all other methods.