llm-runtime-metrics (Python)

Python bindings for request metrics and Prometheus export.

Install from PyPI as llm-runtime-metrics.

Import in Python as:

import llm_runtime_metrics

The supported package root focuses on the request-metrics workflow.

Add LLM Metrics To An Existing Prometheus Server

from prometheus_client import CollectorRegistry, start_http_server
from llm_runtime_metrics import (
    REQUEST_FEATURE_IMAGE,
    REQUEST_FEATURE_TOOLS,
    RequestMetricsCollector,
    RequestMetricsFactory,
)

# Reuse your existing registry if you already have one.
registry = CollectorRegistry()

factory = RequestMetricsFactory(
    request_log_enabled=False,
    metric_prefix="llm_runtime",
    metrics_window_seconds=60.0,
    metrics_quantiles=[0.5, 0.9, 0.99],
)

# Registers a custom collector that pulls fresh samples from `factory` at scrape time.
RequestMetricsCollector(
    factory,
    base_labels={"service": "text-generation", "engine": "vllm"},
    registry=registry,
)

# If your app already exposes /metrics, wire this into that server instead.
start_http_server(8000, registry=registry)


# Example lifecycle hooks in your inference code:
def on_request_start(prompt_token_ids: list[int]):
    features = REQUEST_FEATURE_TOOLS | REQUEST_FEATURE_IMAGE
    return factory.new_request(prompt_token_ids, features=features)


def on_stream_step(req_metrics, full_output_token_ids: list[int], cached_tokens: int | None):
    # Use `is_diff=False` when passing cumulative token ids.
    req_metrics.record_tokens(full_output_token_ids, cached_tokens=cached_tokens, is_diff=False)


def on_request_success(req_metrics):
    req_metrics.success()


def on_request_cancel(req_metrics):
    req_metrics.cancel()

The exported request latency distributions include request_metrics_ttft_ms, request_metrics_time_to_fifth_token_ms, request_metrics_first_to_fifth_token_itl_ms, and request_metrics_itl_ms. request_metrics_time_to_fifth_token_ms is the elapsed time from request start until the fifth output token is observed by record_tokens. request_metrics_first_to_fifth_token_itl_ms is the elapsed time between observing the first and fifth output tokens.

The inflight gauges also include early-decode pressure between the first and fifth output token: current_inflight_first_to_fifth_token_requests, mean_inflight_first_to_fifth_token_requests, current_inflight_first_to_fifth_token_input_tokens, and mean_inflight_first_to_fifth_token_input_tokens. The input-token gauges count prompt tokens attached to requests that have emitted at least one output token but have not emitted their fifth token yet.

Available request feature bits:

REQUEST_FEATURE_NONE
REQUEST_FEATURE_XGRAMMAR
REQUEST_FEATURE_TOOLS
REQUEST_FEATURE_IMAGE

If you need plain text output instead of a collector, call:

text = factory.prometheus_strfmt({"service": "text-generation", "engine": "vllm"})

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

llm_runtime_metrics-0.0.16.tar.gz (98.7 kB view details)

Uploaded Jul 23, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

llm_runtime_metrics-0.0.16-cp314-cp314t-musllinux_1_2_x86_64.whl (7.2 MB view details)

Uploaded Jul 23, 2026 CPython 3.14tmusllinux: musl 1.2+ x86-64

llm_runtime_metrics-0.0.16-cp314-cp314t-musllinux_1_2_i686.whl (7.2 MB view details)

Uploaded Jul 23, 2026 CPython 3.14tmusllinux: musl 1.2+ i686

llm_runtime_metrics-0.0.16-cp314-cp314t-musllinux_1_2_armv7l.whl (6.9 MB view details)

Uploaded Jul 23, 2026 CPython 3.14tmusllinux: musl 1.2+ ARMv7l

llm_runtime_metrics-0.0.16-cp314-cp314t-musllinux_1_2_aarch64.whl (7.0 MB view details)

Uploaded Jul 23, 2026 CPython 3.14tmusllinux: musl 1.2+ ARM64

llm_runtime_metrics-0.0.16-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.6 MB view details)

Uploaded Jul 23, 2026 CPython 3.14tmanylinux: glibc 2.17+ x86-64

llm_runtime_metrics-0.0.16-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (8.3 MB view details)

Uploaded Jul 23, 2026 CPython 3.14tmanylinux: glibc 2.17+ ppc64le

llm_runtime_metrics-0.0.16-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl (7.3 MB view details)

Uploaded Jul 23, 2026 CPython 3.14tmanylinux: glibc 2.17+ i686

llm_runtime_metrics-0.0.16-cp314-cp314t-macosx_11_0_arm64.whl (5.8 MB view details)

Uploaded Jul 23, 2026 CPython 3.14tmacOS 11.0+ ARM64

llm_runtime_metrics-0.0.16-cp314-cp314t-macosx_10_12_x86_64.whl (6.1 MB view details)

Uploaded Jul 23, 2026 CPython 3.14tmacOS 10.12+ x86-64

llm_runtime_metrics-0.0.16-cp310-abi3-win_amd64.whl (5.6 MB view details)

Uploaded Jul 23, 2026 CPython 3.10+Windows x86-64

llm_runtime_metrics-0.0.16-cp310-abi3-musllinux_1_2_x86_64.whl (7.2 MB view details)

Uploaded Jul 23, 2026 CPython 3.10+musllinux: musl 1.2+ x86-64

llm_runtime_metrics-0.0.16-cp310-abi3-musllinux_1_2_i686.whl (7.2 MB view details)

Uploaded Jul 23, 2026 CPython 3.10+musllinux: musl 1.2+ i686

llm_runtime_metrics-0.0.16-cp310-abi3-musllinux_1_2_armv7l.whl (6.9 MB view details)

Uploaded Jul 23, 2026 CPython 3.10+musllinux: musl 1.2+ ARMv7l

llm_runtime_metrics-0.0.16-cp310-abi3-musllinux_1_2_aarch64.whl (7.0 MB view details)

Uploaded Jul 23, 2026 CPython 3.10+musllinux: musl 1.2+ ARM64

llm_runtime_metrics-0.0.16-cp310-abi3-manylinux_2_28_ppc64le.whl (8.3 MB view details)

Uploaded Jul 23, 2026 CPython 3.10+manylinux: glibc 2.28+ ppc64le

llm_runtime_metrics-0.0.16-cp310-abi3-manylinux_2_28_armv7l.whl (6.6 MB view details)

Uploaded Jul 23, 2026 CPython 3.10+manylinux: glibc 2.28+ ARMv7l

llm_runtime_metrics-0.0.16-cp310-abi3-manylinux_2_28_aarch64.whl (6.8 MB view details)

Uploaded Jul 23, 2026 CPython 3.10+manylinux: glibc 2.28+ ARM64

llm_runtime_metrics-0.0.16-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.6 MB view details)

Uploaded Jul 23, 2026 CPython 3.10+manylinux: glibc 2.17+ x86-64

llm_runtime_metrics-0.0.16-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl (7.3 MB view details)

Uploaded Jul 23, 2026 CPython 3.10+manylinux: glibc 2.17+ i686

llm_runtime_metrics-0.0.16-cp310-abi3-macosx_11_0_arm64.whl (5.8 MB view details)

Uploaded Jul 23, 2026 CPython 3.10+macOS 11.0+ ARM64

llm_runtime_metrics-0.0.16-cp310-abi3-macosx_10_12_x86_64.whl (6.1 MB view details)

Uploaded Jul 23, 2026 CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file llm_runtime_metrics-0.0.16.tar.gz.

File metadata

Download URL: llm_runtime_metrics-0.0.16.tar.gz
Upload date: Jul 23, 2026
Size: 98.7 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.14.1

File hashes

Hashes for llm_runtime_metrics-0.0.16.tar.gz
Algorithm	Hash digest
SHA256	`1ebbbd1eca9d1f0a13f910c8093aa772ebcb9c1a969f6c954bdf7a4420586f16`
MD5	`556f8ab5cb5fff9b569ec7bd676efbf4`
BLAKE2b-256	`72c8e26a30495d89e62d16dbdba96d56a32c2b53161e0e39fda4b4a6905da48a`

Algorithm	Hash digest
SHA256	`52dde57e9645230a7f59285891ece46afc573fca3c532ee1c68411bb89fbf58b`
MD5	`8d8662d9710b6267fd225c1addb8278b`
BLAKE2b-256	`ad99cb2a8fcfef3f0748aaffe1278b290fd46a347148d6630f6ff5fa1a26b38e`

Algorithm	Hash digest
SHA256	`f8efa94fec119d4fc8880bfa28a343b93a0f4f53bb00eda0ef682ebe8c80e585`
MD5	`e61744b9ab92eb9c625ecb87f11ea3a0`
BLAKE2b-256	`7c0513e641a503503992e5f9c15f74ed7e76183c69e5062435428cc55505b211`

Algorithm	Hash digest
SHA256	`bf431261e51b83244c4fb13747c2b15fbe4357c146a123e4d5cd46e5d3d07416`
MD5	`955ae7b6ac7596ccbd8d3fea9c8c85b8`
BLAKE2b-256	`abef0c9182ea64c78d0d21899b5a079bd492b01ad6126dc802aff9b64e0150d0`

Algorithm	Hash digest
SHA256	`bc42b9036ed005a47bb5f8522d4e25914b7712ba3ad07b81b0589254045d113b`
MD5	`b9c0710c8d4dfd742cbfafbdd274ca99`
BLAKE2b-256	`29301a608bba1f514e8a6f07ebaff8e1f545d5270531fc83516e0b2ffcffbd51`

Algorithm	Hash digest
SHA256	`5f4a7df2c650b279d03fc64a9de35cea88e1f045adece05f60b0406a67114f74`
MD5	`15687588117c21ce845c393cf364f9a3`
BLAKE2b-256	`f4d015699c5b728e801cd3d4cf47283045c23f3cde8c6f1f880483d14a9e33eb`

Algorithm	Hash digest
SHA256	`c1e3d23e640dedf270a2a4c18c4a6f68ac8f6637ea9dde3c9cf559247ea560fa`
MD5	`1ab1aa2f95c76ea137e343d191dd994a`
BLAKE2b-256	`dba2c470c86788c416deb0cf523c1fbd5b9bb20e5c8f827a54580d59955144d8`

Algorithm	Hash digest
SHA256	`a341b8f1ad495d8fef598ad076102c3bdf5e0c9ac0577bf308218535bb80e75b`
MD5	`6e1d3d1c38c27b009a49b5f4a7f92b5d`
BLAKE2b-256	`376c3b8d10cf0208546f5a949546b8b1cee958bf3311ff929831358be5c21f11`

Algorithm	Hash digest
SHA256	`ff1e2a5573da36e139aa4c331e5b12f3882822f2d83a3b2c4bd841ba58f672f2`
MD5	`8397bb3098662e384dcca3cd5ebc4053`
BLAKE2b-256	`b95f14ceb4363640f877c290771167b9f3e349fcbfdc9129c58ed57d4e1f316e`

Algorithm	Hash digest
SHA256	`d1c487507908c142af6a55678abdfd496704aa182a87331def1732b75ffa1d72`
MD5	`8cd8d4867bf841c2b2640aef8b5cfb70`
BLAKE2b-256	`2895ebb25f6774982e952b6638f399195c5cac3120209c5e2e37f964376b0548`

Algorithm	Hash digest
SHA256	`364592af9e0d2ed20fa30e026733a0618503bbf6c824428e209730f943ae0b93`
MD5	`24054764dd9d5b5fe9e0239bb0e0cbb0`
BLAKE2b-256	`78eab8191488ff0e82a660e1fe7f0e56d4c6e3adaca01e90a438e2f522b42669`

Algorithm	Hash digest
SHA256	`e1fcd27abdcf068188f33d6f3c822ec09cc91738128e84d11e9c1c63bbcb33f3`
MD5	`bc8f8f8fb122683ba84351e642977ba5`
BLAKE2b-256	`1d40dea03f3bb9b2eb969d63a8e09bcf92f5e0a154364477f3909aebc8ec0775`

Algorithm	Hash digest
SHA256	`c08010294c60910d0e757e018ddeeaccf0a453fe3e0214643b77f99ed91b2af0`
MD5	`040b36b89b25e960a93d9b0bbb120c3d`
BLAKE2b-256	`78340f9324a6caa2dbc548e7a049e99b316bf2fdfef249eb1ec50e0af6966c4b`

Algorithm	Hash digest
SHA256	`b9ed3102f527c959106a1ff18deb20285bb58464413f50172c77d6706b794abd`
MD5	`3ea202c277c48a4074faed16a9b8fa6e`
BLAKE2b-256	`b83bbffd9328ba63ccf74fbf01bd844442aad8f0b908aa77eb57a22ca73875c2`

Algorithm	Hash digest
SHA256	`fe22fce7b81bbb2527e3c879741ebf3a6f9d6b61bcc30bfbe6576525388a448f`
MD5	`c87043c701f968f91738fef5d2d020b3`
BLAKE2b-256	`309dad917927796262dc845c557b8b274712fe328fe1e29207e053aaedd5675f`

Algorithm	Hash digest
SHA256	`78279424cadf5a0815530cc41bd508b218110396bad74c7d494286feca5ceb32`
MD5	`502b560ba04544965a0aa9ad886e6a8e`
BLAKE2b-256	`dcb9b7f1d659d5b934570a80c0dd0b466ea623c9ee08ebce0b1965378a36464d`

Algorithm	Hash digest
SHA256	`9c0f888cfc22fa17018ff44024cd7a8902566b1469abeea186b0727f66e3a859`
MD5	`2418ffaa1c2d8b79f819f92f57961e47`
BLAKE2b-256	`83e14b1d01945df2dc9f0e53c9ef3087e3b7449f71d2d9ddae2c5d700d502ad4`

Algorithm	Hash digest
SHA256	`d8d2c4179cbb043828a0f2136fc722ed9b5117839ddef8f8e9df1d069d5ba0a0`
MD5	`acacd23859d427f7cd76b1aaebab48ad`
BLAKE2b-256	`267de8074e212819e67630a7789f52045684a7dc96fd28bc8a7ccbbc5885e6ef`

Algorithm	Hash digest
SHA256	`1bde53c22672ea0b25ec0a4f1ec03bb92e601ee5316d1b0099d71d0af04aac09`
MD5	`3317469e21d490755440ac1872e2b0d8`
BLAKE2b-256	`8efa821f7827d68f5a153e3d3f331780add890e6724732714d48ba1684ffc331`

Algorithm	Hash digest
SHA256	`0a0f3180e8e13f845039cbd122a58f6f69e387a061824350c7cac572559b404e`
MD5	`0098c20b79aa886a8048cf0a8c48c73d`
BLAKE2b-256	`d7ad8b092f1745b12cf432be550475e4b813360ff4275d7c0a23c37293d9f88e`

Algorithm	Hash digest
SHA256	`5e001541be966ea34d6c9c680f8d82e07543026b730f2ba8f49e8127633e9109`
MD5	`4312126331796f6c40e1b9f67271a30f`
BLAKE2b-256	`d8b60af9c3527ea41abf37bf3c1c5869ffacf5b32d5be2d7a3ed6c1ca859b91f`

llm-runtime-metrics 0.0.16

llm-runtime-metrics (Python)

Add LLM Metrics To An Existing Prometheus Server

Project links

Key dates

1 maintainer

Credits

License

Requires

Tags

Classifiers

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

Algorithm	Hash digest
SHA256	`ebead30069ad444f2680c0c3beb1f6501e77adc3f87667bc887fa1c8d853910f`
MD5	`9d8e3ba4faa6b7b0e69f93acbfc105a4`
BLAKE2b-256	`dac5f7bd306df93db0251af4f0ccd4b08871f47c98b364e223799ccdeeb1cc07`