veval-sdk

Python SDK for Veval — trace, evaluate, and test AI agents.

Install

pip install veval-sdk

Quick start

Wrap your agent with run_async to send a trace to the Veval dashboard:

import asyncio
from veval import VevalSdk, VevalOptions, VevalExecutionContext

sdk = VevalSdk(VevalOptions(api_key="veval-..."))

async def my_agent(ctx: VevalExecutionContext) -> str:
    result = await ctx.track_step_async(
        name="call-llm",
        input=ctx.input,
        step=lambda: call_my_llm(ctx.input),
    )
    return result

output = asyncio.run(sdk.run_async("my-agent", my_agent, input="Hello"))

Every call to track_step_async records the step name, input, output, timing, and any metadata you attach.

Tracking steps

Use the StepHandle to attach token counts, cost, and model name:

async def agent(ctx: VevalExecutionContext) -> str:
    async def llm_call(handle):
        response = await call_my_llm(ctx.input)
        handle.set_meta("tokens_in", response.usage.input_tokens)
        handle.set_meta("tokens_out", response.usage.output_tokens)
        handle.set_meta("cost_usd", response.usage.cost)
        handle.set_meta("model", "claude-opus-4-6")
        return response.text

    return await ctx.track_step_async("call-llm", ctx.input, llm_call)

Assertions

Use TraceAssert to validate behaviour at the trace level:

from veval import TraceAssert

assertions = [
    TraceAssert.no_errors(),
    TraceAssert.max_steps(10),
    TraceAssert.step_exists("call-llm"),
    TraceAssert.max_cost(0.05),
    TraceAssert.max_duration(30_000),
    TraceAssert.output_contains("success"),
    TraceAssert.tool_called("web-search"),
]

Scenarios

Run a named scenario against a list of test items and assertions:

from veval import ScenarioItem

items = [
    ScenarioItem(name="basic greeting", input="Hello"),
    ScenarioItem(name="edge case", input=""),
]

result = asyncio.run(
    sdk.run_scenario_async("smoke-test", my_agent, assertions, items=items)
)

print(f"Passed: {result.pass_count}/{len(result.results)}")
for r in result.results:
    if not r.passed:
        print(f"  FAIL {r.item.name}: {r.failures}")

You can also pull items from the Veval dashboard by omitting items:

result = asyncio.run(sdk.run_scenario_async("smoke-test", my_agent, assertions))

Replay and snapshot testing

Load a previously recorded trace and replay it with mocked LLM responses — no real calls, deterministic results:

trace = asyncio.run(sdk.get_trace_async("tr_abc123"))

replay = asyncio.run(sdk.replay_async(
    trace,
    my_agent,
    ReplayOptions(mock_llm_responses=True, assertions=assertions),
))

print("Passed" if replay.passed else replay.failures)

Store a known-good run — every step with its input and output — and detect when a new run drifts from it: a changed prompt or tool argument, a step added, dropped, repeated, or reordered:

# Save a baseline from a recorded trace. The trace is pinned, so retention never deletes it.
asyncio.run(sdk.save_snapshot_async("my-baseline", "tr_abc123"))

# In tests: an assertion like any other. A missing baseline fails; it never passes silently.
replay = asyncio.run(sdk.replay_async(trace, my_agent, ReplayOptions(
    mock_llm_responses=True,
    assertions=[TraceAssert.matches_snapshot(sdk, "my-baseline")],
    compare_with_recording=SnapshotOptions(),  # also fail if steps/inputs drift from the trace
)))

# Or compare directly, and record the result in the dashboard.
baseline = asyncio.run(sdk.get_snapshot_async("my-baseline"))
diff = asyncio.run(sdk.compare_snapshot_async("my-baseline", baseline, ctx))
if diff.has_changes:
    print(diff.summary())  # every change, with a line diff of changed inputs

Test SDK

VevalTestSdk is a drop-in replacement that blocks real LLM calls, making it safe to use in unit tests. It still reports to the dashboard.

from veval import VevalTestSdk

sdk = VevalTestSdk(options).with_replay(trace)
output = asyncio.run(sdk.run_async("my-agent", my_agent, input="test input"))

API reference

VevalOptions

VevalSdk

TraceAssert built-ins

Custom assertions implement async ITraceAssertion.evaluate_async(ctx) -> Optional[str] — return None to pass or an error string to fail.

Requirements

• Python 3.10+
• No third-party dependencies