vcti-benchmark
Benchmarking for the vcti Python packages: the quantities worth measuring about running code, the instruments that take them, and the convention for running one from a script.
The model is vcti-measure's
— measurements, instruments, observations, studies and one JSON artifact per
run. This package is what makes that model about software performance. It
adds no concepts.
Installation
pip install vcti-benchmark
In pyproject.toml dependencies
[project.optional-dependencies]
bench = ["vcti-benchmark>=2.0.0"]
Quick Start
A benchmark is a study whose items are callables. benchmark() declares one:
from vcti.benchmark import ALLOCATION, UNITS, WALL_CLOCK_TIME, benchmark, run_script
from vcti.measure.core import Axis, Manifest
MANIFEST = Manifest(
"vcti.nputils", # your namespace, so results never collide
[
benchmark(
"build",
summary="Building an array of n elements.",
subjects={"growable": lambda values, n: build_growable(values)},
references={"list": lambda values, n: list(values)},
axes=[Axis(name="n", values=(1_000, 100_000), labels=("1K", "100K"))],
setup=lambda n: range(n),
measurements=[WALL_CLOCK_TIME, ALLOCATION],
follow=["wall-clock-time.median", "allocation.net"],
)
],
units=list(UNITS),
)
if __name__ == "__main__":
run_script(MANIFEST, package="vcti-nputils")
That is the whole of benchmarks/bench_build.py. Running it prints each
observation as it lands and writes one JSON artifact into
benchmarks/results/, named for the study, the commit and the artifact's own
identifier — so repeated runs of the same commit accumulate rather than
overwrite, and a name that somehow already exists is reported rather than
replaced:
build/growable n=1000 [wall-clock-time] median=32.4 µs stddev=8.54 µs ops=27.9 K/s
build/growable n=1000 [allocation] peak=31.2 KiB net=31.1 KiB
...
8 measured
wrote benchmarks/results/build-af7822d-5d748a3e91c4402fb0d7f1c88e2a6b35.json
Rendering it is a separate step, in a separate package — so last month's run can be re-rendered without measuring anything again.
What you get
WALL_CLOCK_TIME |
Calibrated per-call time: min, max, mean, median, stddev, iqr, rounds, outliers, ops |
ELAPSED_TIME |
One call, measured once — for work that does not survive repetition |
ALLOCATION |
Transient peak and net bytes, from one act of measuring |
RESIDENT_SET |
Declared; nothing here provides it |
CalibratedTimer, SingleShotTimer, TracedAllocation |
The instruments, standard library only |
UNITS |
ns, bytes, count, ratio, per-second, with their display ladders |
benchmark() |
Declares a study whose items are callables |
run_script() |
Describe the run, run it, write the artifact, say where |
conditions(), source(), environment() |
Which build, on which machine |
A few things worth knowing
setupcannot see which implementation is about to run. It takes the axis values and nothing else, so every implementation is handed equivalent input by construction rather than by care.- Each measurement is taken separately, with its own preparation. Traced allocation inside a timing loop would distort the timings and report a peak accumulated across thousands of calls.
- An attempt that produced nothing still appears, carrying a status and a
reason. Asking for
RESIDENT_SETgives you observations marked unavailable saying so, not silence. netis notretained. It is allocated minus freed, which is not a reachability analysis — an operation returning a view over existing memory allocates nothing.- The timer records how it ran — whether collection was suspended, how many warmup rounds it discarded, and how many calls each round performed — as factors beside the numbers. A median averaged over one call and one averaged over four thousand are not equally trustworthy.
- Which instrument answered is provenance, never identity, so replacing one does not orphan the series it fed.
Dependencies
vcti-measure. The instruments themselves need only the standard library.
Documentation
| If you want to… | Read |
|---|---|
| Understand the methodology and what is still unsettled | docs/design.md |
| Understand the measurement model itself | vcti-measure's design document |
Metadata
Release files for vcti-benchmark 2.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| vcti_benchmark-2.0.0.tar.gz | 38.9 kB | With provenance | Details for vcti_benchmark-2.0.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| vcti_benchmark-2.0.0-py3-none-any.whl | Python 3 | none | any | Details for vcti_benchmark-2.0.0-py3-none-any.whl |
Total release size: 63.9 kB
Release files / vcti_benchmark-2.0.0.tar.gz
| Download URL | vcti_benchmark-2.0.0.tar.gz |
|---|---|
| Size | 38.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1df4f68df894b7cf5a02d93550fa3ace8620969d8e7ee8da2eeea3109624f2b9
|
|
BLAKE2b-256 checksum How to use checksums |
5f8bda7cbb8af7592ed0f1e9c238cdfd6a2ee7eb416486f197c3bf43e08272e4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 5, 2026.
Transparency logRelease files / vcti_benchmark-2.0.0-py3-none-any.whl
| Download URL | vcti_benchmark-2.0.0-py3-none-any.whl |
|---|---|
| Size | 25.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
03c5f9521265d6e5a17f4611e8d890fde2b970693bff4364edb44ab863ef35d3
|
|
BLAKE2b-256 checksum How to use checksums |
1094970ddf991e45aaec4e77fdf101004b5b6c14a9f905b801b1f09e6f1a573c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 5, 2026.
Transparency logSecurity for vcti-benchmark 2.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for vcti_benchmark-2.0.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 5, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.