polarity-keystone 0.2.0

Python SDK client for the Keystone agent evaluation + sandboxed-execution platform

Keystone SDK for Python

Python client for the Keystone agent evaluation + sandboxed-execution platform. Shares a single source of truth with the TypeScript and Go SDKs — byte-identical cost estimates and prompt rendering across all three.

Install

pip install polarity-keystone

Zero external runtime dependencies — uses only urllib from the standard library.

Quick start

from polarity_keystone import (
    Keystone, traced,
    Factuality, ExactMatch, FileExists,
)

ks = Keystone(api_key="ks_live_...")

# Create an experiment and run it with three client-side scorers.
exp = ks.experiments.create("nightly-regression", spec_id="spec-123")
results = ks.experiments.run_and_wait(
    exp.id,
    scores=[
        Factuality(model="paragon-fast"),
        ExactMatch(expected_key="expected", case_sensitive=False),
        FileExists("output.txt", gate=True),
    ],
)
print(f"pass rate: {results.metrics.pass_rate:.0%}")

# Stream every trace for the experiment.
for trace in ks.export.traces(experiment_id=exp.id):
    print(trace["tool"], trace["cost"])

# Wrap an OpenAI/Anthropic client for auto-trace ingest
from openai import OpenAI
client = ks.wrap(OpenAI(), sandbox_id="sb-xxx")
client.chat.completions.create(model="gpt-4o", messages=[...])  # auto-reports

What's in the SDK

• 9 client services — sandboxes, specs, experiments, alerts, agents, datasets, scoring, export, prompts
• 28 built-in scorers across 5 families:
  • Heuristic (6): ExactMatch, Levenshtein, NumericDiff, JSONDiff, JSONValidity, SemanticListContains
  • LLM-judge (9): Factuality, Battle, ClosedQA, Humor, Moderation, Summarization, SQLJudge, Translation, Security
  • RAG (8): ContextPrecision, ContextRecall, ContextRelevancy, ContextEntityRecall, Faithfulness, AnswerRelevancy, AnswerSimilarity, AnswerCorrectness
  • Embedding (1): EmbeddingSimilarity
  • Sandbox invariants (5): FileExists, FileContains, CommandExits, SQLEquals, LLMJudge
• @Scorer decorator — wrap any (scenario) -> score function
• @traced decorator + context manager — auto-capture spans with AsyncLocalStorage-equivalent context propagation
• Client wrapping — ks.wrap(openai_client, sandbox_id=...) adds transparent trace reporting for OpenAI + Anthropic (sync / async / streaming)
• auto_instrument() — single call patches every installed provider (OpenAI, Anthropic, Mistral, Google GenAI, LiteLLM, Claude Agent SDK, DSPy, LangChain)
• Prompt management — ks.prompts.create(slug, template) / ks.prompts.get(slug) / Prompt.render(**vars) with server-side audit + byte-identical local renderer
• Bulk export — ks.export.traces(...), .spans(...), .scenarios(...), .scores(...) auto-paginate; ks.export.experiment(id) for full JSON or NDJSON dumps
• OpenTelemetry bridge — wrap() emits gen_ai.* metadata; register_otel_flush() piggy-backs on program shutdown

Versioning

Semver. 0.x = alpha/beta while the API shape settles.

License

MIT.