polarity-keystone 0.3.1

Python SDK for the Polarity agent evaluation + sandboxed-execution platform (legacy package name: keystone)

Polarity SDK for Python

Python client for the Polarity agent evaluation + sandboxed-execution platform. Shares a single source of truth with the TypeScript and Go SDKs — byte-identical cost estimates and prompt rendering across all three.

Rebrand note: this SDK is now called Polarity. The previous import name (polarity_keystone) and class name (Keystone) continue to work indefinitely, so existing code does not need to change. New code should prefer from polarity import Polarity.

Install

pip install polarity-keystone

The pip package name is unchanged; only the public import name changed. Zero external runtime dependencies — uses only urllib from the standard library.

60-second quick start: Eval()

The shortest path from "I have an agent" to "I have an evaluation":

from polarity import Eval, Factuality, AnswerRelevancy

result = Eval(
    "summarisation-quality",
    data=[
        {"input": "Long article about whales...", "expected": "Whales are mammals."},
        {"input": "Article about Java GC...",     "expected": "Java GC reclaims memory."},
    ],
    task=lambda input: my_agent(input),                # your agent / prompt
    scores=[
        Factuality(model="paragon-fast"),
        AnswerRelevancy(),
    ],
    max_concurrency=4,
)

print(result.summary)                                   # p50/p95/mean per scorer

If POLARITY_API_KEY (or the legacy KEYSTONE_API_KEY) is set, the run is also recorded to your dashboard; otherwise it stays purely local.

Build a dataset from production traces

Turn failing prod traces into a regression eval set in one call:

from polarity import Polarity

plr = Polarity()
result = plr.datasets.from_traces(
    name="summarize-regressions",
    filter={"tool": "summarize", "since": "24h"},
)
print(result["rows_added"], result["sample_rows"])

Idempotent on span_id within a single call. Events missing input / expected are skipped and counted under result["skipped"]. Pass dry_run=True to inspect the rows without writing.

Pytest plugin: eval-as-tests

Once the package is installed, pytest auto-loads a polarity_score fixture (the legacy keystone_score name keeps working):

from polarity.scorers import Factuality

def test_summary(polarity_score):
    out = my_agent("Long text…")
    polarity_score(Factuality(expected="Short summary"),
                   input="Long text…", output=out, min=0.8)

If min is set and the score falls below it, the test fails. At session end the plugin posts one eval per test module to the dashboard (POLARITY_API_KEY required; pass --polarity-no-flush — or the legacy --keystone-no-flush — to skip the post).

Sandbox-as-a-tool ergonomics

create() / get() / list() return a bound SandboxHandle so an agent loop can call the sandbox without threading the ID:

sb = plr.sandboxes.create(spec_id="spec-123")

sb.exec("python script.py")
sb.write("/tmp/input.json", json.dumps(payload))
out = sb.read("/tmp/output.json")
diff = sb.diff()

sb.destroy()

Same pattern on ExperimentHandle and AgentSnapshotHandle:

exp = plr.experiments.create("nightly", spec_id="s")
results = exp.run_and_wait(scores=[Factuality(), ExactMatch(expected_key="expected")])
cmp = exp.compare(other_exp)                            # handle or string ID
m   = exp.metrics()

snap = plr.agents.upload("codex", "./bundle", entrypoint=["python3", "agent.py"])
snap.delete()

Auto-instrument every LLM client at once

from polarity import auto_instrument

auto_instrument(sandbox_id=os.environ.get("POLARITY_SANDBOX_ID"))
# Patches every installed provider in-place. Subsequent OpenAI/Anthropic/
# Mistral/Google/LiteLLM/Claude Agent SDK/DSPy/LangChain calls auto-trace.

Wraps OpenAI, Anthropic, Mistral, Google GenAI, LiteLLM, Claude Agent SDK, DSPy, LangChain in one call — every prompt, token count, and tool call shows up in your dashboard with no other code changes.

Manual tracing when you want it

from polarity import traced

# 1. Decorator
@traced
def fetch_user(uid: str) -> dict:
    return db.users.find(uid)

# 2. Named decorator
@traced(name="planning-phase")
def plan(prompt: str) -> str:
    return llm.complete(prompt)

# 3. Context manager
with traced(name="embed-doc"):
    openai.embeddings.create(...)

Spans automatically nest using contextvar-based parent linking — no need to plumb a context object through your code.

Environment variables

Both the new Polarity-branded names and the legacy Keystone-branded names are honored; the new name wins when both are set.

Preferred	Legacy (still works)	Purpose
POLARITY_API_KEY	KEYSTONE_API_KEY	API key for authentication
POLARITY_BASE_URL	KEYSTONE_BASE_URL	Override the default server URL
POLARITY_SANDBOX_ID	KEYSTONE_SANDBOX_ID	Injected by the platform inside sandboxes

Default server URL: https://plr.sh.

API keys begin with plr_live_… going forward; existing ks_live_… keys continue to be accepted by the server.

What's in the SDK

• 9 client services — sandboxes, specs, experiments, alerts, agents, datasets, scoring, export, prompts
• 3 bound handles — SandboxHandle, ExperimentHandle, AgentSnapshotHandle with delegated methods
• 29 built-in scorers across 5 families:
  • Heuristic (6): ExactMatch, Levenshtein, NumericDiff, JSONDiff, JSONValidity, SemanticListContains
  • LLM-judge (9): Factuality, Battle, ClosedQA, Humor, Moderation, Summarization, SQLJudge, Translation, Security
  • RAG (8): ContextPrecision, ContextRecall, ContextRelevancy, ContextEntityRecall, Faithfulness, AnswerRelevancy, AnswerSimilarity, AnswerCorrectness
  • Embedding (1): EmbeddingSimilarity
  • Sandbox invariants (5): FileExists, FileContains, CommandExits, SQLEquals, LLMJudge
• @Scorer decorator — wrap any (scenario) -> score function
• Eval(name, data, task, scores) — Braintrust-style one-call eval primitive
• @traced decorator + context manager — auto-capture spans with contextvar-based propagation
• Client wrapping — plr.wrap(openai_client, sandbox_id=...) adds transparent trace reporting (sync / async / streaming)
• auto_instrument() — single call patches every installed provider (OpenAI, Anthropic, Mistral, Google GenAI, LiteLLM, Claude Agent SDK, DSPy, LangChain)
• Prompt management — plr.prompts.create(slug, template) / plr.prompts.get(slug) / Prompt.render(**vars) with server-side audit + byte-identical local renderer
• Bulk export — plr.export.traces(...), .spans(...), .scenarios(...), .scores(...) auto-paginate; plr.export.experiment(id) for full JSON or NDJSON dumps
• OpenTelemetry bridge — wrap() emits gen_ai.* metadata; register_otel_flush() piggy-backs on program shutdown

Versioning

Semver. 0.x = alpha/beta while the API shape settles.

License

MIT.