polarity-keystone 0.2.5

Python SDK client for the Keystone agent evaluation + sandboxed-execution platform

Keystone SDK for Python

Python client for the Keystone agent evaluation + sandboxed-execution platform. Shares a single source of truth with the TypeScript and Go SDKs — byte-identical cost estimates and prompt rendering across all three.

Install

pip install polarity-keystone

Zero external runtime dependencies — uses only urllib from the standard library.

60-second quick start: Eval()

The shortest path from "I have an agent" to "I have an evaluation":

from polarity_keystone import Eval, Factuality, AnswerRelevancy

result = Eval(
    "summarisation-quality",
    data=[
        {"input": "Long article about whales...", "expected": "Whales are mammals."},
        {"input": "Article about Java GC...",     "expected": "Java GC reclaims memory."},
    ],
    task=lambda input: my_agent(input),                # your agent / prompt
    scores=[
        Factuality(model="paragon-fast"),
        AnswerRelevancy(),
    ],
    max_concurrency=4,
)

print(result.summary)                                   # p50/p95/mean per scorer

If KEYSTONE_API_KEY is set, the run is also recorded to your dashboard; otherwise it stays purely local. Same shape in TypeScript and Go.

Sandbox-as-a-tool ergonomics

create() / get() / list() return a bound SandboxHandle so an agent loop can call the sandbox without threading the ID:

sb = ks.sandboxes.create(spec_id="spec-123")

sb.exec("python script.py")
sb.write("/tmp/input.json", json.dumps(payload))
out = sb.read("/tmp/output.json")
diff = sb.diff()

sb.destroy()

Same pattern on ExperimentHandle and AgentSnapshotHandle:

exp = ks.experiments.create("nightly", spec_id="s")
results = exp.run_and_wait(scores=[Factuality(), ExactMatch(expected_key="expected")])
cmp = exp.compare(other_exp)                            # handle or string ID
m   = exp.metrics()

snap = ks.agents.upload("codex", "./bundle", entrypoint=["python3", "agent.py"])
snap.delete()

The handles delegate field access to the underlying dataclass, so reading sb.id, exp.status, snap.version keeps working unchanged. The old service-level methods (ks.sandboxes.run_command(id, …), ks.experiments.run(id)) stay too — handle methods just call them.

Auto-instrument every LLM client at once

from polarity_keystone import auto_instrument

auto_instrument(sandbox_id=os.environ.get("KEYSTONE_SANDBOX_ID"))
# Patches every installed provider in-place. Subsequent OpenAI/Anthropic/
# Mistral/Google/LiteLLM/Claude Agent SDK/DSPy/LangChain calls auto-trace.

Wraps OpenAI, Anthropic, Mistral, Google GenAI, LiteLLM, Claude Agent SDK, DSPy, LangChain in one call — every prompt, token count, and tool call shows up in your dashboard with no other code changes.

Manual tracing when you want it

from polarity_keystone import traced

# 1. Decorator
@traced
def fetch_user(uid: str) -> dict:
    return db.users.find(uid)

# 2. Named decorator
@traced(name="planning-phase")
def plan(prompt: str) -> str:
    return llm.complete(prompt)

# 3. Context manager
with traced(name="embed-doc"):
    openai.embeddings.create(...)

Spans automatically nest using contextvar-based parent linking — no need to plumb a context object through your code.

What's in the SDK

• 9 client services — sandboxes, specs, experiments, alerts, agents, datasets, scoring, export, prompts
• 3 bound handles — SandboxHandle, ExperimentHandle, AgentSnapshotHandle with delegated methods
• 29 built-in scorers across 5 families:
  • Heuristic (6): ExactMatch, Levenshtein, NumericDiff, JSONDiff, JSONValidity, SemanticListContains
  • LLM-judge (9): Factuality, Battle, ClosedQA, Humor, Moderation, Summarization, SQLJudge, Translation, Security
  • RAG (8): ContextPrecision, ContextRecall, ContextRelevancy, ContextEntityRecall, Faithfulness, AnswerRelevancy, AnswerSimilarity, AnswerCorrectness
  • Embedding (1): EmbeddingSimilarity
  • Sandbox invariants (5): FileExists, FileContains, CommandExits, SQLEquals, LLMJudge
• @Scorer decorator — wrap any (scenario) -> score function
• Eval(name, data, task, scores) — Braintrust-style one-call eval primitive
• @traced decorator + context manager — auto-capture spans with contextvar-based propagation
• Client wrapping — ks.wrap(openai_client, sandbox_id=...) adds transparent trace reporting (sync / async / streaming)
• auto_instrument() — single call patches every installed provider (OpenAI, Anthropic, Mistral, Google GenAI, LiteLLM, Claude Agent SDK, DSPy, LangChain)
• Prompt management — ks.prompts.create(slug, template) / ks.prompts.get(slug) / Prompt.render(**vars) with server-side audit + byte-identical local renderer
• Bulk export — ks.export.traces(...), .spans(...), .scenarios(...), .scores(...) auto-paginate; ks.export.experiment(id) for full JSON or NDJSON dumps
• OpenTelemetry bridge — wrap() emits gen_ai.* metadata; register_otel_flush() piggy-backs on program shutdown

Versioning

Semver. 0.x = alpha/beta while the API shape settles.

License

MIT.