llm-runtime-metrics (Python)

Python bindings for request metrics and Prometheus export.

Install from PyPI as llm-runtime-metrics.

Import in Python as:

import llm_runtime_metrics

The supported package root focuses on the request-metrics workflow.

Add LLM Metrics To An Existing Prometheus Server

from prometheus_client import CollectorRegistry, start_http_server
from llm_runtime_metrics import (
    REQUEST_FEATURE_IMAGE,
    REQUEST_FEATURE_TOOLS,
    RequestMetricsCollector,
    RequestMetricsFactory,
)

# Reuse your existing registry if you already have one.
registry = CollectorRegistry()

factory = RequestMetricsFactory(
    request_log_enabled=False,
    metric_prefix="llm_runtime",
    metrics_window_seconds=60.0,
    metrics_quantiles=[0.5, 0.9, 0.99],
)

# Registers a custom collector that pulls fresh samples from `factory` at scrape time.
RequestMetricsCollector(
    factory,
    base_labels={"service": "text-generation", "engine": "vllm"},
    registry=registry,
)

# If your app already exposes /metrics, wire this into that server instead.
start_http_server(8000, registry=registry)


# Example lifecycle hooks in your inference code:
def on_request_start(prompt_token_ids: list[int]):
    features = REQUEST_FEATURE_TOOLS | REQUEST_FEATURE_IMAGE
    return factory.new_request(prompt_token_ids, features=features)


def on_stream_step(req_metrics, full_output_token_ids: list[int], cached_tokens: int | None):
    # Use `is_diff=False` when passing cumulative token ids.
    req_metrics.record_tokens(full_output_token_ids, cached_tokens=cached_tokens, is_diff=False)


def on_request_success(req_metrics):
    req_metrics.success()


def on_request_cancel(req_metrics):
    req_metrics.cancel()

The exported request latency distributions include request_metrics_ttft_ms, request_metrics_time_to_fifth_token_ms, request_metrics_first_to_fifth_token_itl_ms, and request_metrics_itl_ms. request_metrics_time_to_fifth_token_ms is the elapsed time from request start until the fifth output token is observed by record_tokens. request_metrics_first_to_fifth_token_itl_ms is the elapsed time between observing the first and fifth output tokens.

The inflight gauges also include early-decode pressure between the first and fifth output token: current_inflight_first_to_fifth_token_requests, mean_inflight_first_to_fifth_token_requests, current_inflight_first_to_fifth_token_input_tokens, and mean_inflight_first_to_fifth_token_input_tokens. The input-token gauges count prompt tokens attached to requests that have emitted at least one output token but have not emitted their fifth token yet.

Available request feature bits:

  • REQUEST_FEATURE_NONE
  • REQUEST_FEATURE_XGRAMMAR
  • REQUEST_FEATURE_TOOLS
  • REQUEST_FEATURE_IMAGE

If you need plain text output instead of a collector, call:

text = factory.prometheus_strfmt({"service": "text-generation", "engine": "vllm"})

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.0.16)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for michaelfeil from gravatar.com michaelfeil

Credits

Author: Baseten

License

Apache Software License (Apache-2.0)

Requires

Python >=3.10

Tags

metrics observability prometheus llm

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

llm_runtime_metrics-0.0.14.tar.gz (98.2 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

llm_runtime_metrics-0.0.14-cp314-cp314t-musllinux_1_2_x86_64.whl (7.2 MB view details)

Uploaded CPython 3.14tmusllinux: musl 1.2+ x86-64

llm_runtime_metrics-0.0.14-cp314-cp314t-musllinux_1_2_i686.whl (7.2 MB view details)

Uploaded CPython 3.14tmusllinux: musl 1.2+ i686

llm_runtime_metrics-0.0.14-cp314-cp314t-musllinux_1_2_armv7l.whl (6.9 MB view details)

Uploaded CPython 3.14tmusllinux: musl 1.2+ ARMv7l

llm_runtime_metrics-0.0.14-cp314-cp314t-musllinux_1_2_aarch64.whl (7.0 MB view details)

Uploaded CPython 3.14tmusllinux: musl 1.2+ ARM64

llm_runtime_metrics-0.0.14-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.6 MB view details)

Uploaded CPython 3.14tmanylinux: glibc 2.17+ x86-64

llm_runtime_metrics-0.0.14-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (8.3 MB view details)

Uploaded CPython 3.14tmanylinux: glibc 2.17+ ppc64le

llm_runtime_metrics-0.0.14-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl (7.3 MB view details)

Uploaded CPython 3.14tmanylinux: glibc 2.17+ i686

llm_runtime_metrics-0.0.14-cp314-cp314t-macosx_11_0_arm64.whl (6.0 MB view details)

Uploaded CPython 3.14tmacOS 11.0+ ARM64

llm_runtime_metrics-0.0.14-cp314-cp314t-macosx_10_12_x86_64.whl (6.3 MB view details)

Uploaded CPython 3.14tmacOS 10.12+ x86-64

llm_runtime_metrics-0.0.14-cp310-abi3-win_amd64.whl (5.7 MB view details)

Uploaded CPython 3.10+Windows x86-64

llm_runtime_metrics-0.0.14-cp310-abi3-musllinux_1_2_x86_64.whl (7.2 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ x86-64

llm_runtime_metrics-0.0.14-cp310-abi3-musllinux_1_2_i686.whl (7.2 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ i686

llm_runtime_metrics-0.0.14-cp310-abi3-musllinux_1_2_armv7l.whl (6.9 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ ARMv7l

llm_runtime_metrics-0.0.14-cp310-abi3-musllinux_1_2_aarch64.whl (7.0 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ ARM64

llm_runtime_metrics-0.0.14-cp310-abi3-manylinux_2_28_ppc64le.whl (8.3 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.28+ ppc64le

llm_runtime_metrics-0.0.14-cp310-abi3-manylinux_2_28_armv7l.whl (6.7 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.28+ ARMv7l

llm_runtime_metrics-0.0.14-cp310-abi3-manylinux_2_28_aarch64.whl (6.8 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.28+ ARM64

llm_runtime_metrics-0.0.14-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.6 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

llm_runtime_metrics-0.0.14-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl (7.3 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ i686

llm_runtime_metrics-0.0.14-cp310-abi3-macosx_11_0_arm64.whl (6.0 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

llm_runtime_metrics-0.0.14-cp310-abi3-macosx_10_12_x86_64.whl (6.3 MB view details)

Uploaded CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file llm_runtime_metrics-0.0.14.tar.gz.

File metadata

  • Download URL: llm_runtime_metrics-0.0.14.tar.gz
  • Upload date:
  • Size: 98.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: maturin/1.14.1

File hashes

Hashes for llm_runtime_metrics-0.0.14.tar.gz
Algorithm Hash digest
SHA256 889fc878add39b573df0ab2097a2030c73396e453a47d874a84306e9a6c6f9be
MD5 70b4bcb8e9d1a5cb5359211a6c8cded6
BLAKE2b-256 52d2c20d853d70355721de5050e13e22850d588dd6bc0b12eb9226bcfa0669aa

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp314-cp314t-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp314-cp314t-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 eca78e3e58c4e894b304c61648dd811c1475cfa9a710d3ef33ff9e848c7e5216
MD5 8e0b69840539b4891cbfe595edc34b55
BLAKE2b-256 b0d05ae08cf6b40e6f3b2b3b055f53d61cd41dc2c054739cf3cd8badef8ea82d

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp314-cp314t-musllinux_1_2_i686.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp314-cp314t-musllinux_1_2_i686.whl
Algorithm Hash digest
SHA256 d3f4cc266f400273fe48cd4b32cb37d638825eb5f75c39d31fc2ae898740d9d5
MD5 f86e8548e41cad087aa008993bd78d61
BLAKE2b-256 4c9a5b57b705e4f27296bff0b24406233eb3eac1396a367150aa6d6cc0581288

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp314-cp314t-musllinux_1_2_armv7l.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp314-cp314t-musllinux_1_2_armv7l.whl
Algorithm Hash digest
SHA256 232da15447346979dc7530e866e720bd8328d3521be056131f809e8f527faffc
MD5 78ccee2a471c881f776a55d1389fc607
BLAKE2b-256 cb33a05b6bc2b83bd4919e61725b4ecd05d297720ecbc471ec732239324fdf46

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp314-cp314t-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp314-cp314t-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 71eb18f9449a6866b787c9d8914f70b2d86ec63fe4a54c8a2cb3f3fb9cbfe99d
MD5 56877e0810c4acc49427e92d0de260de
BLAKE2b-256 1bc372e0bbd26c1c359932d1df29d030bd02e571c6e617bfbc41c3aa4e57f96d

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 e756f5fab5d101b1cf229f83acda936ee6a80552569fdfed7bb3c7fd3fdc4f23
MD5 001e5e3ead0cb7c4cee8fb7bef9dcb36
BLAKE2b-256 04c13d942b82bb61c09d6215175e4f2a46374781d9b8318b9cc2946f7aa790f3

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 8603b04d8a16950cb51981505ea9d77aaa9b89520e38a9c26a3f7d7872919909
MD5 1f5d5f0f926f61d9a3c2ffe053d419b9
BLAKE2b-256 4ea25134ee6660271fcc6a686c6bb347f91f97b74a1b56cfe31b3e265f975d54

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm Hash digest
SHA256 a5f629574ab02533d494c08f29baa4721e2e3a38462630fb5049a2c5d2638470
MD5 23bc9a0316f9eb7b5e8212c10fa48aff
BLAKE2b-256 387115176b191cc4f305b33007ea4a6d2fb42bcd7a5cc079a8b85d9a0ec7c709

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp314-cp314t-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp314-cp314t-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 4ecf76726dc31c4b2f3321afd246351a6999edbe633cae30aae7d822ab07c88e
MD5 87958a2242c5f9d717e2d669792d1e9e
BLAKE2b-256 344c9410a5ff84f36ca249abeb8db139e548f04c0e3f8bbad7581478deac9a7c

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp314-cp314t-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp314-cp314t-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 a0095242116d3f2a7edaec9f0bf3121c5b78d1e0c7fb8cac1a4db91cf24e5029
MD5 4fc26ba7ce28d2901087da0e6718ad59
BLAKE2b-256 9a83759545ee86876149a74dd89d6a3b878972df956996ee397bc2ed971659fc

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp310-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 2481153d37c5ae83aca71e305013b661c3f7400c1acb5b6147e87c3232ef1e90
MD5 f2d6980c2fa34af7dcd936ed4f58f0b9
BLAKE2b-256 9bf4bbddd09404ca6260a586285b811f6575f6470ef6ff8272fdd61f5b518309

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp310-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp310-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 39248291792e2a4af1e70058a4ab7d4b4987ad316139a9e7cf9cc43a4a407ac1
MD5 2466555dae39ca17c6b3e32420961049
BLAKE2b-256 573a8106d86092afb697dd63945154f45569c417db4c11fefbf2333ef5bdbd69

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp310-abi3-musllinux_1_2_i686.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp310-abi3-musllinux_1_2_i686.whl
Algorithm Hash digest
SHA256 7ab8a42e6fc97d554d2ac7dfb06c0ae210c3868b7040820b5a303f22b937be66
MD5 34f5e9901bb67a0badcedbb0c04cb909
BLAKE2b-256 722b151ba91e5e1b0e71d9c11ab33182daecb23ec885ca1c4cf51c88c2f280c8

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp310-abi3-musllinux_1_2_armv7l.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp310-abi3-musllinux_1_2_armv7l.whl
Algorithm Hash digest
SHA256 4edd124f524fead3cfdaa0f355ff7c9fbb8abcca7d7c7c28d3bf90faa83b8779
MD5 c8dbcd6f71f1aec24d381c1ce3452358
BLAKE2b-256 d76efc1845b8c45955d6954d3ea19b8b6ef1a4191b4f53f5129b56101e8b2179

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp310-abi3-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp310-abi3-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 768a0a6307fed3f627f894752b6ee6deb4c8df8400d1ec08c1b46c8cac0b8f26
MD5 4cb910dffd262dc53813b9dc051bf770
BLAKE2b-256 55505dabbb9fb37475b87e61a14f5a1ad672f3eea91f93cb47e8c10726fe24a8

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp310-abi3-manylinux_2_28_ppc64le.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp310-abi3-manylinux_2_28_ppc64le.whl
Algorithm Hash digest
SHA256 93024dee78279a101f8fc225bffc2cf72534699aa63e82d923200b3be10921b1
MD5 fcee49e3fb075ea1966d28021fa0cfb3
BLAKE2b-256 551b7932587cd5444d2f429daf3afa9e0e911c30a6858325bb22f0b516a05f26

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp310-abi3-manylinux_2_28_armv7l.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp310-abi3-manylinux_2_28_armv7l.whl
Algorithm Hash digest
SHA256 579fad5baede51e1b0c1d7f017cc932c43b0452f9e27be0ecde7f9312e011b07
MD5 5923676ce6c31f279a6d9b85430ecc1b
BLAKE2b-256 0a1b28933d362fb882a9e0e83e97f39df55737114777ffea8848a6a7333420d4

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp310-abi3-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp310-abi3-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 a61d188c178320beb58ec8d44f5f041324213c0f12495f350711cdeec4efde8a
MD5 b1f9ef6d379ab24a222fe974d7032319
BLAKE2b-256 eae701a7975a98ff2f7aec432165c97f67b1946925b57755790b5f8a3baf3834

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 211afa4f22a2dce9b6e7e10b08f568018b9ca1b5759938ce4ffba8c214687283
MD5 5fa8b8c4a3cfdc4432b40ba25e89d330
BLAKE2b-256 0c3bd45c9d968b66b911585142689065bcc28b2d748357c252c1dc11f4a5a822

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm Hash digest
SHA256 e56491cfb57d3636cdd08422b1be86385af1a64de24f32250d61a935aa2147e1
MD5 16470ced3c645068c4ccf3f63dfa29fe
BLAKE2b-256 89aee95aada3a73379a8bc40c3a81134d37e7c6ca7e1d8bdebab83627e7e3979

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 ee4c1a9e8187f7522f486b70f505e07a4e4ea921db1f420d499efc1cad5d3ba6
MD5 413e6c592821abfdf37955c2174a5c09
BLAKE2b-256 8f6d68fe6b3ab2c74012e69ab943ca4a5f26b55dec3e2b835f32ec15d4473cb4

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.14-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.14-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 8be82abde299fe9e709a0530215ee31b2ca426a80911b749b21a543d4bc4afbf
MD5 cbdc6ab42fed3e501ee5c3f383b4e407
BLAKE2b-256 d143dd5f47dea9e0aaad7f6f27b3976055a288a079297269b6da823236ec3cc8

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page