Skip to main content

krite-bench

Benchmark harness for any engine that serves POST /v1/systemone (Krite Protocol v1). It measures quality, calibration, option-order invariance, question interference, latency, and memory under docs/benchmark-spec.md. It talks to engines only over HTTP and contains no ML code.

pip install krite-bench

Workspace

Every path resolves against a workspace with the Krite repository layout: $KRITE_ROOT, else the krite checkout the package sits in, else the current directory.

Path Content
benchmarks/baselines/engines.toml engines to measure (you write it)
benchmarks/data/ suites built by krite-bench data (dataset text stays local)
benchmarks/results/baselines/ result rows (--out to change)

An engine needs a name and a port; start it yourself before measuring:

[krite-v1]
port = 8147
model = "krite-0.15b-v1"
backend = "candle-metal"
precision = "fp32"
start = "krite serve --port 8147 --model /path/to/krite-0.15b-v1"

Commands

krite-bench env                                   # environment record (required before measuring)
krite-bench data [--only agnews-choice] [--verify]
krite-bench quality --engine krite-v1 [--langs en] [--memory]
krite-bench calibrate --engine krite-v1
krite-bench invariance --engine krite-v1 [--n 100]
krite-bench interference --engine krite-v1
krite-bench latency --engines krite-v1 [--mode burst|sustained] [--cells primary|aux|matrix] [--cache cold|warm]
krite-bench report --dest baselines.md

latency --cache startup and encoder start processes through the repository's benchmarks/baselines/serve.sh and target/release/krite; they need a checkout.

License: Apache-2.0.

Metadata

Release files for krite-bench 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for krite-bench 0.1.0
File Size Uploaded Provenance Actions
krite_bench-0.1.0.tar.gz 24.3 kB Without provenance Details for krite_bench-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for krite-bench 0.1.0
File Interpreter ABI Platform Actions
krite_bench-0.1.0-py3-none-any.whl Python 3 none any Details for krite_bench-0.1.0-py3-none-any.whl

Total release size: 53.6 kB

Release files / krite_bench-0.1.0.tar.gz

Download URL krite_bench-0.1.0.tar.gz
Size 24.3 kB
Tags Source
SHA-256 checksum
How to use checksums
433477ebfa9501f020e62b6edb70878b489fbb556655684360758aa1d51cdffb
BLAKE2b-256 checksum
How to use checksums
9cf5375e17f755212e19c1c99dd6d7ac195584ab44386c29458aaad3381c2688
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / krite_bench-0.1.0-py3-none-any.whl

Download URL krite_bench-0.1.0-py3-none-any.whl
Size 29.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
44dce1c36bf3d75f9eb5e740ccc922908470013d547f9e502bea16a94b6ee11e
BLAKE2b-256 checksum
How to use checksums
1d7fc6625e2eb404283716cd7be4f23d64186b799b7a4c3bdf0b6335681a85a3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for krite-bench 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page