Skip to main content

vcti-benchmark

Benchmarking for the vcti Python packages: the quantities worth measuring about running code, the instruments that take them, and the convention for running one from a script.

The model is vcti-measure's — measurements, instruments, observations, studies and one JSON artifact per run. This package is what makes that model about software performance. It adds no concepts.

Installation

pip install vcti-benchmark

In pyproject.toml dependencies

[project.optional-dependencies]
bench = ["vcti-benchmark>=2.0.0"]

Quick Start

A benchmark is a study whose items are callables. benchmark() declares one:

from vcti.benchmark import ALLOCATION, UNITS, WALL_CLOCK_TIME, benchmark, run_script
from vcti.measure.core import Axis, Manifest

MANIFEST = Manifest(
    "vcti.nputils",                       # your namespace, so results never collide
    [
        benchmark(
            "build",
            summary="Building an array of n elements.",
            subjects={"growable": lambda values, n: build_growable(values)},
            references={"list": lambda values, n: list(values)},
            axes=[Axis(name="n", values=(1_000, 100_000), labels=("1K", "100K"))],
            setup=lambda n: range(n),
            measurements=[WALL_CLOCK_TIME, ALLOCATION],
            follow=["wall-clock-time.median", "allocation.net"],
        )
    ],
    units=list(UNITS),
)

if __name__ == "__main__":
    run_script(MANIFEST, package="vcti-nputils")

That is the whole of benchmarks/bench_build.py. Running it prints each observation as it lands and writes one JSON artifact into benchmarks/results/, named for the study, the commit and the artifact's own identifier — so repeated runs of the same commit accumulate rather than overwrite, and a name that somehow already exists is reported rather than replaced:

  build/growable n=1000 [wall-clock-time]  median=32.4 µs  stddev=8.54 µs  ops=27.9 K/s
  build/growable n=1000 [allocation]       peak=31.2 KiB  net=31.1 KiB
  ...
  8 measured
  wrote benchmarks/results/build-af7822d-5d748a3e91c4402fb0d7f1c88e2a6b35.json

Rendering it is a separate step, in a separate package — so last month's run can be re-rendered without measuring anything again.


What you get

WALL_CLOCK_TIME Calibrated per-call time: min, max, mean, median, stddev, iqr, rounds, outliers, ops
ELAPSED_TIME One call, measured once — for work that does not survive repetition
ALLOCATION Transient peak and net bytes, from one act of measuring
RESIDENT_SET Declared; nothing here provides it
CalibratedTimer, SingleShotTimer, TracedAllocation The instruments, standard library only
UNITS ns, bytes, count, ratio, per-second, with their display ladders
benchmark() Declares a study whose items are callables
run_script() Describe the run, run it, write the artifact, say where
conditions(), source(), environment() Which build, on which machine

A few things worth knowing

  • setup cannot see which implementation is about to run. It takes the axis values and nothing else, so every implementation is handed equivalent input by construction rather than by care.
  • Each measurement is taken separately, with its own preparation. Traced allocation inside a timing loop would distort the timings and report a peak accumulated across thousands of calls.
  • An attempt that produced nothing still appears, carrying a status and a reason. Asking for RESIDENT_SET gives you observations marked unavailable saying so, not silence.
  • net is not retained. It is allocated minus freed, which is not a reachability analysis — an operation returning a view over existing memory allocates nothing.
  • The timer records how it ran — whether collection was suspended, how many warmup rounds it discarded, and how many calls each round performed — as factors beside the numbers. A median averaged over one call and one averaged over four thousand are not equally trustworthy.
  • Which instrument answered is provenance, never identity, so replacing one does not orphan the series it fed.

Dependencies

vcti-measure. The instruments themselves need only the standard library.


Documentation

If you want to… Read
Understand the methodology and what is still unsettled docs/design.md
Understand the measurement model itself vcti-measure's design document

Metadata

Release files for vcti-benchmark 2.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for vcti-benchmark 2.0.0
File Size Uploaded Provenance Actions
vcti_benchmark-2.0.0.tar.gz 38.9 kB With provenance Details for vcti_benchmark-2.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for vcti-benchmark 2.0.0
File Interpreter ABI Platform Actions
vcti_benchmark-2.0.0-py3-none-any.whl Python 3 none any Details for vcti_benchmark-2.0.0-py3-none-any.whl

Total release size: 63.9 kB

Release files / vcti_benchmark-2.0.0.tar.gz

Download URL vcti_benchmark-2.0.0.tar.gz
Size 38.9 kB
Tags Source
SHA-256 checksum
How to use checksums
1df4f68df894b7cf5a02d93550fa3ace8620969d8e7ee8da2eeea3109624f2b9
BLAKE2b-256 checksum
How to use checksums
5f8bda7cbb8af7592ed0f1e9c238cdfd6a2ee7eb416486f197c3bf43e08272e4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 5, 2026.

Transparency log

Release files / vcti_benchmark-2.0.0-py3-none-any.whl

Download URL vcti_benchmark-2.0.0-py3-none-any.whl
Size 25.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
03c5f9521265d6e5a17f4611e8d890fde2b970693bff4364edb44ab863ef35d3
BLAKE2b-256 checksum
How to use checksums
1094970ddf991e45aaec4e77fdf101004b5b6c14a9f905b801b1f09e6f1a573c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 5, 2026.

Transparency log

Security for vcti-benchmark 2.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for vcti_benchmark-2.0.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 5, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

2.0.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page