metrik-profiler

Measured latency and memory for transformer models — gated by what the device can actually measure.

Part of Metrik. Install the umbrella distribution (metrik-ai) rather than this package directly unless you are building against it.

The problem this package is shaped around

Most machines cannot measure most of what a profile can express. A CPU-only laptop has no VRAM timeline, no power counter, and no measured FLOP count. The tempting design is a struct of optional floats where an unmeasurable field quietly settles at 0.0 — and a reader cannot tell zero because idle from zero because unavailable.

So the mechanism is inverted. A capability probe records what this device was confirmed to read; every measurement names the capability it came from; and ProfileRecord rejects any measurement citing a capability that was not probed. Fabricating a number is a validation error rather than a matter of discipline.

$ metrik profile --check
backend  cpu
counter source  none

capability      state          detail
wall_clock      measurable     perf_counter around a synchronous backend
peak_allocated  not measurable host RAM has no allocator watermark
nvml_power      not measurable no NVIDIA device is present
flops_measured  not measurable no kernel-level counter source

A profile that measured one thing is a good artifact. A profile that measured one thing and implies four is the failure this exists to prevent.

Two statistics decisions

The interval is bootstrapped on the median, not computed on the mean. Latency is right-skewed — a slow sample can be arbitrarily slow, a fast one is bounded by the work — so the mean sits above the median and a t-interval around it can exclude the median entirely. Since the record reports a median, an interval around the mean would bound a statistic that is not there.

Below 5 samples there is no interval at all. A 95% interval of unstated construction over three points is decoration, and the schema refuses to carry one.

Install

pip install metrik-profiler            # capability probe and refusal path; no ML stack
pip install "metrik-profiler[torch]"   # adds torch and transformers, enough to measure

The split is deliberate: metrik profile --check answers "what can this machine measure?" without an ML stack, which is exactly the machine where the question is worth asking.

With an NVIDIA GPU, the extra is not enough. pip install torch resolves to a CPU-only build on Windows and Linux, which imports fine and never touches the GPU. Add the CUDA index:

pip install torch --index-url https://download.pytorch.org/whl/cu126

metrik profile --check diagnoses this case specifically rather than reporting the generic "no CUDA device", because that wording reads like a hardware fault when the hardware is fine.

License

Apache-2.0