dvbench client

The dvbench Python client creates named experiments, tails a live ZKDV records.cborl transcript, and sends each complete CBOR item in a CBOR transport envelope. Records above the shared chunk threshold use multiple acknowledged messages and are reassembled before worker validation. The client does not interpret or validate records. Record shape, cryptographic, and protocol validation are entirely the server's responsibility.

The process uses placeholder user_id zero and generates a client_id at startup. Each record_id is the BLAKE3 digest of the exact encoded record bytes, and the first record's ID is the stable transcript_id. Record events are numbered from zero. On SIGINT or SIGTERM, the client sends a Close event whose sequence is the next expected value and waits for its acknowledgement. A failed record send is retried in memory; restarting the client replays the transcript under a new client_id.

The public lifecycle is:

from dvbench import Client

experiment = Client().experiment("owner/model")
proof = experiment.proof(backend="jax")
# Train through the ordinary ZKDV API, then close the proof before measuring.
proof.finish()

measurement = experiment.measure(checkpoint)

@measurement.evaluate(params=0, batch=1)
def evaluate(params, batch):
    return score(params, batch)

@measurement.aggregate
def aggregate(outputs):
    return sum(outputs) / len(outputs)

measurement.submit("accuracy", measurement.run(checkpoint, batches))

Measurement definitions and benchmarks use the same client rather than a separate HTTP integration:

accuracy = client.define_measurement("accuracy", "Held-out exact-match accuracy.")
benchmark = client.create_benchmark(
    "Core evaluations",
    "Primary held-out evaluation suite.",
    [accuracy["id"]],
    accuracy["id"],
)

Checkpoint stores and mock generators that already have the compact inner Merkle root can use experiment.measure_opening(inner_root). If an experiment has more than one run, pass the intended transcript_id. The projected leaves are never sent to the API or retained in SQLite.

measure(checkpoint) downloads the selected run's probe-vector object in chunks, verifies and caches its object hash for the client lifetime, then projects and hashes one leaf at a time. The worker accepts the compact opening only when it matches an attested update boundary. Evaluation outputs remain submitter-reported; the client signs their trace but does not attest their execution.

measurement.run(...) serializes each complete input batch and output tensor as canonical, content-addressed CBOR. measurement.submit(...) streams those objects to the API before sending the small signed trace of digest/length references. The worker, rather than the client, derives sample alignment from the batch dimension. Uploads are bounded-concurrent and content-deduplicated for the client session. Temporary client objects are removed after submission.

The native uploader uses these environment variables when configured directly:

• ZKDV_RECORDS_PATH — transcript path; defaults to ../zkdv/proof/records.cborl when run from the dvbench root.
• DVBENCH_STREAM_URL — WebSocket destination.
• DVBENCH_CLIENT_POLL_MS — file polling interval; defaults to 100.
• DVBENCH_CLIENT_RECONNECT_MS — reconnect delay; defaults to 1000.

Python development

Build and install the mixed Python/Rust package into its project environment with:

uv sync --all-extras --dev

The Maturin backend declared in pyproject.toml builds and installs the native uploader and protocol helpers as dvbench._core.

Build the distributable source archive and platform wheel with the standard PEP 517 path:

uv build

Documentation

Build or preview the Zensical site through the project environment:

uv run --group docs zensical build --strict
uv run --group docs zensical serve