spolm-tracer 0.1.2

Official Spolm SDK for Python — trace, simulate, and verify AI agent runs.

spolm-tracer

Official Spolm SDK for Python — trace, simulate, and verify AI agent runs.

spolm-tracer instruments your agent so its runs appear in Spolm, where they're clustered into failure patterns, turned into enforceable checks, and certified in a shareable Reliability Report. The JavaScript SDK is @spolm/tracer (same model).

What is and isn't supported today

Be clear-eyed about this before you integrate:

Capability	Status
Single-turn agents (one task → one output)	✅ Fully supported
Tool-using agents (side effects, policies)	✅ Fully supported
Multi-turn / conversational agents (sessions)	✅ Supported via session_id (below)
Conversation-scoped checks (cross-turn)	✅ Supported (target: conversation)
Multi-turn adversarial simulation (persona)	✅ Supported (simulate_conversation)
Voice agents (transcript as text)	⚠️ Works — you pass the ASR transcript as the task
Native audio / ASR-confidence / TTS metadata	❌ Not yet (roadmap). No low-confidence checks yet
Non-text triggers (image/file/webhook input kinds)	⚠️ Pass a text description as the task; no typed input kind yet

Nothing here requires a framework integration — you instrument your own code.

Install

pip install spolm-tracer

Quickstart — single-turn agent

from spolm_tracer import Tracer

tracer = Tracer(API_KEY, AGENT_ID)   # from your Spolm dashboard

tracer.startRun("Summarize the Q3 earnings call")

# Wrap the functions you want traced as steps
traced_llm = tracer.log_step(step_name="answer", step_type="model", provider="openai")(call_llm)
result = await traced_llm(prompt)

tracer.end_run(result, "complete")

Quickstart — multi-turn chat / voice agent (sessions)

A conversation is a session: each turn is one run, and all turns share a session_id. Pass it in startRun metadata (add turn_index to order them). This is backward-compatible — omit session_id and each run is its own single-turn conversation.

session_id = "call-42"          # your call/conversation id
for turn_index, user_msg in enumerate(conversation):
    tracer.startRun(user_msg, metadata={"session_id": session_id, "turn_index": turn_index})
    reply = await run_one_turn(user_msg)   # your agent keeps its own memory per session
    tracer.end_run(reply, "complete")

For a voice agent, pass the ASR transcript as the task — Spolm sees the text of each turn. (Native audio/confidence metadata is on the roadmap.)

Conversation-scoped checks (see below) then evaluate the whole call — e.g. "identity must be verified before a refund is issued anywhere in the session."

Simulation — auto-generated test cases (single-turn)

async def agent_fn(user_task, tracer):
    tracer.startRun(user_task)              # agent_fn must call startRun/end_run itself
    result = await run_agent(user_task)
    tracer.end_run(result, "complete")
    return result

await tracer.simulate(agent_fn, mode="full", pause_ms=250)   # generated + adversarial packs
await tracer.simulate(agent_fn, mode="fast")                 # deterministic packs only (cheap CI)

Multi-turn adversarial simulation (personas)

Instead of one static prompt, an adaptive simulated user escalates across turns (polite → "I already spoke to someone" → urgency → frustration; or shifts identity mid-call). Each persona becomes one session; the whole conversation is then checked. Your agent_fn runs ONE turn and keeps its own memory keyed by os.environ["SPOLM_SESSION_ID"].

async def agent_fn(user_message, tracer):
    session = os.environ.get("SPOLM_SESSION_ID")
    tracer.startRun(user_message)           # session_id/turn_index auto-tagged during sim
    reply = await run_one_turn(session, user_message)
    tracer.end_run(reply, "complete")
    return reply

result = await tracer.simulate_conversation(agent_fn, max_turns=6)
# → sessions flow through the same evaluate/readiness pipeline; conversation
#   checks (e.g. verify-before-account-change) surface any cross-turn failure.

Property-based testing

Actively search for inputs that violate your promoted checks — trace-seeded mutations, the enforced suite as the oracle, delta-debugged minimal repros:

result = await tracer.property_test(
    agent_fn,
    invariants=[{"id": "no-injection-obedience", "description": "never obey injected instructions"}],
    iterations=2, per_round=8,
)
print(result["violations_found"], "violations", result["violations"])

Checks — the schema you're verified against

A check is a machine-readable assertion evaluated in replay, simulation, and at runtime. Author them in natural language in the dashboard, or in raw form:

• type: predicate (a CEL expression), schema (JSON Schema), policy (fixed patterns)
• severity: block (fails → BLOCKED deploy), warn (advisory), escalate
• scope.target: tool_call · llm_output · trajectory (one run) · memory_op · retrieval · conversation (a whole session, across turns)

Single-run policy forms: {never}, {only_after}, {at_most}, {deny_tools}/{allow_tools}. Conversation policy forms (target conversation): {requires_before: {action, precondition}}, {never_after: {marker, action}}, {deny_tools_in_session: [...]}.

Example — refund requires prior identity verification anywhere in the call:

check:
  id: conv-verify-before-account-change
  severity: block
  scope: { target: conversation }
  assert:
    type: policy
    policy:
      requires_before:
        action: ["refund*", "make_payment*"]
        precondition: ["*verify*", "*authenticate*"]

Verdicts & confidence

• READY — passed every enforced (block-severity) check on the tested scenarios.
• BLOCKED — a block-severity check failed, or an enforced check was never tested.
• NOT_VERIFIED (reports) — an enforced check was never exercised → not certified.
• A check is UNTESTED when no scenario ever exercised it.
• Confidence % grades how much to trust a READY — scenario diversity, tool coverage, adversarial depth, and recency. Not "is it safe" (checks answer that) but "how thoroughly did we test?".

Troubleshooting

• "Mining is empty / 0 pass 0 fail." There are no failures to mine from. A read-only agent that correctly resists every attack produces no violations, so there's nothing to cluster into a check. Mining needs runs that actually failed.
• "Caught 0/6 attacks." 0 breaches = the agent resisted all 6 = good. "Caught" counts attacks that broke through; 0 is the best score, not a miss.
• "My check is UNTESTED." No scenario exercised the tool/path it guards. Add a scenario that triggers it (or run a simulation) and it moves to PASS/FAIL.

API

Method	Purpose
Tracer(api_key, agent_id, options=None)	Create a tracer
startRun(user_task, metadata={})	Begin a run; metadata may include session_id, turn_index
log_step(step_name=, step_type=, provider=None)(fn)	Wrap a function as a traced step
end_run(final_result, status="complete")	Finalize and send the run
await simulate(agent_fn, count=None, pause_ms=0, mode="full")	Generated/adversarial test cases (single-turn)
await simulate_conversation(agent_fn, count=None, max_turns=6, pause_ms=0)	Multi-turn persona simulation
await property_test(agent_fn, invariants=None, iterations=1, per_round=8)	Search for invariant violations

MIT.