veval-sdk

Python SDK for Veval — trace, evaluate, and test AI agents.

Install

pip install veval-sdk

Quick start

Wrap your agent with run_async to send a trace to the Veval dashboard:

import asyncio
from veval import VevalSdk, VevalOptions, VevalExecutionContext

sdk = VevalSdk(VevalOptions(api_key="veval-..."))

async def my_agent(ctx: VevalExecutionContext) -> str:
    result = await ctx.track_step_async(
        name="call-llm",
        input=ctx.input,
        step=lambda: call_my_llm(ctx.input),
    )
    return result

output = asyncio.run(sdk.run_async("my-agent", my_agent, input="Hello"))

Every call to track_step_async records the step name, input, output, timing, and any metadata you attach.

Tracking steps

Use the StepHandle to attach token counts, cost, and model name:

async def agent(ctx: VevalExecutionContext) -> str:
    async def llm_call(handle):
        response = await call_my_llm(ctx.input)
        handle.set_meta("tokens_in", response.usage.input_tokens)
        handle.set_meta("tokens_out", response.usage.output_tokens)
        handle.set_meta("cost_usd", response.usage.cost)
        handle.set_meta("type", "llm")
        handle.set_meta("model", "claude-opus-4-6")
        return response.text

    return await ctx.track_step_async("call-llm", ctx.input, llm_call)

Assertions

Use TraceAssert to validate behaviour at the trace level:

from veval import TraceAssert

assertions = [
    TraceAssert.no_errors(),
    TraceAssert.max_steps(10),
    TraceAssert.step_exists("call-llm"),
    TraceAssert.max_cost(0.05),
    TraceAssert.max_duration(30_000),
    TraceAssert.output_contains("success"),
    TraceAssert.tool_called("web-search"),
]

Scenarios

Run a named scenario against a list of test items and assertions:

from veval import ScenarioItem

items = [
    ScenarioItem(name="basic greeting", input="Hello"),
    ScenarioItem(name="edge case", input=""),
]

result = asyncio.run(
    sdk.run_scenario_async("smoke-test", my_agent, assertions, items=items)
)

print(f"Passed: {result.pass_count}/{len(result.results)}")
for r in result.results:
    if not r.passed:
        print(f"  FAIL {r.item.name}: {r.failures}")

A trace-backed item (ScenarioItem(trace_id=...)) replays live: steps recorded with set_meta("type", "llm") call the model with your current prompt, so a prompt regression fails the item, and every other step returns its recorded output, so tools never reach a real system. Set mock_model_steps=True on the item, or pass mock_model_steps=True to run_scenario_async for every item (including ones loaded from the dashboard), for a free, fully mocked replay that checks code changes but can't see prompt changes. VevalTestSdk always replays trace-backed items mocked. ReplayOptions(live_model_steps=True) does the same for replay_async.

You can also pull items from the Veval dashboard by omitting items:

result = asyncio.run(sdk.run_scenario_async("smoke-test", my_agent, assertions))

Replay and snapshot testing

Load a previously recorded trace and replay it with mocked LLM responses — no real calls, deterministic results:

trace = asyncio.run(sdk.get_trace_async("tr_abc123"))

replay = asyncio.run(sdk.replay_async(
    trace,
    my_agent,
    ReplayOptions(mock_llm_responses=True, assertions=assertions),
))

print("Passed" if replay.passed else replay.failures)

Each call gets the recorded step with the same name and an identical input, or else the next unused one in recorded order, reported as diverged in replay.fidelity. A step with no recording raises instead of calling live, and a recorded step the agent never calls fails the replay.

Store a known-good run — every step with its input and output — and detect when a new run drifts from it: a changed prompt or tool argument, a step added, dropped, repeated, or reordered:

# Save a baseline from a recorded trace. The trace is pinned, so retention never deletes it.
asyncio.run(sdk.save_snapshot_async("my-baseline", "tr_abc123"))

# In tests: an assertion like any other. A missing baseline fails; it never passes silently.
replay = asyncio.run(sdk.replay_async(trace, my_agent, ReplayOptions(
    mock_llm_responses=True,
    assertions=[TraceAssert.matches_snapshot(sdk, "my-baseline")],
    compare_with_recording=SnapshotOptions(),  # also fail if steps/inputs drift from the trace
)))

# Or compare directly, and record the result in the dashboard.
baseline = asyncio.run(sdk.get_snapshot_async("my-baseline"))
diff = asyncio.run(sdk.compare_snapshot_async("my-baseline", baseline, ctx))
if diff.has_changes:
    print(diff.summary())  # every change, with a line diff of changed inputs

Test SDK

VevalTestSdk is a drop-in replacement that blocks real LLM calls, making it safe to use in unit tests. It still reports to the dashboard.

from veval import VevalTestSdk

sdk = VevalTestSdk(options).with_replay(trace)
output = asyncio.run(sdk.run_async("my-agent", my_agent, input="test input"))

API reference

VevalOptions

VevalSdk

TraceAssert built-ins

Custom assertions implement async ITraceAssertion.evaluate_async(ctx) -> Optional[str] — return None to pass or an error string to fail.

Requirements

• Python 3.10+
• No third-party dependencies