llm-runtime-metrics (Python)

Python bindings for request metrics and Prometheus export.

Install from PyPI as llm-runtime-metrics.

Import in Python as:

import llm_runtime_metrics

The supported package root focuses on the request-metrics workflow.

Add LLM Metrics To An Existing Prometheus Server

from prometheus_client import CollectorRegistry, start_http_server
from llm_runtime_metrics import (
    REQUEST_FEATURE_IMAGE,
    REQUEST_FEATURE_TOOLS,
    RequestMetricsCollector,
    RequestMetricsFactory,
)

# Reuse your existing registry if you already have one.
registry = CollectorRegistry()

factory = RequestMetricsFactory(
    request_log_enabled=False,
    metric_prefix="llm_runtime",
    metrics_window_seconds=60.0,
    metrics_quantiles=[0.5, 0.9, 0.99],
)

# Registers a custom collector that pulls fresh samples from `factory` at scrape time.
RequestMetricsCollector(
    factory,
    base_labels={"service": "text-generation", "engine": "vllm"},
    registry=registry,
)

# If your app already exposes /metrics, wire this into that server instead.
start_http_server(8000, registry=registry)


# Example lifecycle hooks in your inference code:
def on_request_start(prompt_token_ids: list[int]):
    features = REQUEST_FEATURE_TOOLS | REQUEST_FEATURE_IMAGE
    return factory.new_request(prompt_token_ids, features=features)


def on_stream_step(req_metrics, full_output_token_ids: list[int], cached_tokens: int | None):
    # Use `is_diff=False` when passing cumulative token ids.
    req_metrics.record_tokens(full_output_token_ids, cached_tokens=cached_tokens, is_diff=False)


def on_request_success(req_metrics):
    req_metrics.success()


def on_request_cancel(req_metrics):
    req_metrics.cancel()

The exported request latency distributions include request_metrics_ttft_ms, request_metrics_time_to_fifth_token_ms, request_metrics_first_to_fifth_token_itl_ms, and request_metrics_itl_ms. request_metrics_time_to_fifth_token_ms is the elapsed time from request start until the fifth output token is observed by record_tokens. request_metrics_first_to_fifth_token_itl_ms is the elapsed time between observing the first and fifth output tokens.

The inflight gauges also include early-decode pressure between the first and fifth output token: current_inflight_first_to_fifth_token_requests, mean_inflight_first_to_fifth_token_requests, current_inflight_first_to_fifth_token_input_tokens, and mean_inflight_first_to_fifth_token_input_tokens. The input-token gauges count prompt tokens attached to requests that have emitted at least one output token but have not emitted their fifth token yet.

Available request feature bits:

  • REQUEST_FEATURE_NONE
  • REQUEST_FEATURE_XGRAMMAR
  • REQUEST_FEATURE_TOOLS
  • REQUEST_FEATURE_IMAGE

If you need plain text output instead of a collector, call:

text = factory.prometheus_strfmt({"service": "text-generation", "engine": "vllm"})

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.0.16)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for michaelfeil from gravatar.com michaelfeil

Credits

Author: Baseten

License

Apache Software License (Apache-2.0)

Requires

Python >=3.10

Tags

metrics observability prometheus llm

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

llm_runtime_metrics-0.0.15.tar.gz (98.6 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

llm_runtime_metrics-0.0.15-cp314-cp314t-musllinux_1_2_x86_64.whl (7.2 MB view details)

Uploaded CPython 3.14tmusllinux: musl 1.2+ x86-64

llm_runtime_metrics-0.0.15-cp314-cp314t-musllinux_1_2_i686.whl (7.2 MB view details)

Uploaded CPython 3.14tmusllinux: musl 1.2+ i686

llm_runtime_metrics-0.0.15-cp314-cp314t-musllinux_1_2_armv7l.whl (6.9 MB view details)

Uploaded CPython 3.14tmusllinux: musl 1.2+ ARMv7l

llm_runtime_metrics-0.0.15-cp314-cp314t-musllinux_1_2_aarch64.whl (7.0 MB view details)

Uploaded CPython 3.14tmusllinux: musl 1.2+ ARM64

llm_runtime_metrics-0.0.15-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.6 MB view details)

Uploaded CPython 3.14tmanylinux: glibc 2.17+ x86-64

llm_runtime_metrics-0.0.15-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (8.3 MB view details)

Uploaded CPython 3.14tmanylinux: glibc 2.17+ ppc64le

llm_runtime_metrics-0.0.15-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl (7.3 MB view details)

Uploaded CPython 3.14tmanylinux: glibc 2.17+ i686

llm_runtime_metrics-0.0.15-cp314-cp314t-macosx_11_0_arm64.whl (5.8 MB view details)

Uploaded CPython 3.14tmacOS 11.0+ ARM64

llm_runtime_metrics-0.0.15-cp314-cp314t-macosx_10_12_x86_64.whl (6.1 MB view details)

Uploaded CPython 3.14tmacOS 10.12+ x86-64

llm_runtime_metrics-0.0.15-cp310-abi3-win_amd64.whl (5.6 MB view details)

Uploaded CPython 3.10+Windows x86-64

llm_runtime_metrics-0.0.15-cp310-abi3-musllinux_1_2_x86_64.whl (7.2 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ x86-64

llm_runtime_metrics-0.0.15-cp310-abi3-musllinux_1_2_i686.whl (7.2 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ i686

llm_runtime_metrics-0.0.15-cp310-abi3-musllinux_1_2_armv7l.whl (6.9 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ ARMv7l

llm_runtime_metrics-0.0.15-cp310-abi3-musllinux_1_2_aarch64.whl (7.0 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ ARM64

llm_runtime_metrics-0.0.15-cp310-abi3-manylinux_2_28_ppc64le.whl (8.3 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.28+ ppc64le

llm_runtime_metrics-0.0.15-cp310-abi3-manylinux_2_28_armv7l.whl (6.6 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.28+ ARMv7l

llm_runtime_metrics-0.0.15-cp310-abi3-manylinux_2_28_aarch64.whl (6.8 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.28+ ARM64

llm_runtime_metrics-0.0.15-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.6 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

llm_runtime_metrics-0.0.15-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl (7.3 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ i686

llm_runtime_metrics-0.0.15-cp310-abi3-macosx_11_0_arm64.whl (5.8 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

llm_runtime_metrics-0.0.15-cp310-abi3-macosx_10_12_x86_64.whl (6.1 MB view details)

Uploaded CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file llm_runtime_metrics-0.0.15.tar.gz.

File metadata

  • Download URL: llm_runtime_metrics-0.0.15.tar.gz
  • Upload date:
  • Size: 98.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: maturin/1.14.1

File hashes

Hashes for llm_runtime_metrics-0.0.15.tar.gz
Algorithm Hash digest
SHA256 2820673f12f5bf1ed61923f235fa81165286bf0151dd98012e3d7bdcaf0f8de7
MD5 767fb1b8dc3b552cb39406795b437f40
BLAKE2b-256 459c51a98d0abfae0ef030ae2656ed34932ab8b238ef28e9226ea992892fd7f2

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp314-cp314t-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp314-cp314t-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 b9bb726b19575f3a92bea6e3d6d52b0d9e97c8d5c3b5beaddb33b7fda36ec20d
MD5 10d044652561060837f0d548193870d9
BLAKE2b-256 f73ce6981ad3b90d3de7156bb7c42587151e4c9d6dce7b80eaad63b324caf2cb

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp314-cp314t-musllinux_1_2_i686.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp314-cp314t-musllinux_1_2_i686.whl
Algorithm Hash digest
SHA256 2d98d83c192cc4318e663ff66a2dea21041393ba976da9b160114a5780a937c1
MD5 e265e6c7ab207e0b26740589627b9da7
BLAKE2b-256 644d2cd89f71c1886b00758697cdcdf4a1eb7c128e6927da6bd27570581bd01a

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp314-cp314t-musllinux_1_2_armv7l.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp314-cp314t-musllinux_1_2_armv7l.whl
Algorithm Hash digest
SHA256 1fc6730ce15a2f5a43c3525d434423dddba08836565c6cd0a5139dd411a88704
MD5 e276f94ea1b3acb84143f4a705c95d22
BLAKE2b-256 eeb63914f38e5f008cd05963dabf0d3c4dddb9d8df6a3700ec0cbf116d6003c2

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp314-cp314t-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp314-cp314t-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 dee5ec554cb2fe75566c34d6bc555915cc947f7f371bfbcf1951fd41f457023c
MD5 0f098127d4fd076ca957957bd3708e0d
BLAKE2b-256 eadd0e047f6fa275067ad66f48df0f2bd0b25217788b9219bbcd1faba94bb7da

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 802389c42be875c2a4b96f77d92d9eb8b241dfd29d3c7eab2b36a28f60e35943
MD5 acbd0d7ab30aceabbf2ba2f4896663e6
BLAKE2b-256 5010cd8a2fba93b95ee3754baf82ad4125fc0deada048edc5ae5ccbb2dbb1fb8

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 8a2ce9408d65e50309e1ed171b20b172942ed224c229e107829842e4bf9979ce
MD5 285aa3486f75444b120ae401a21c40b3
BLAKE2b-256 b1238553d9f2471f68cc198b7fafe2bf98da5f9c863e68e4f7a8623664b592a9

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm Hash digest
SHA256 2076df3963c594ef8429750de732ce033c9a80cde51e012c0031191b9a6f7fb9
MD5 68a55a19b110802dc27cd6d27d2cd689
BLAKE2b-256 d229ebd89337b7b7dc12b78cb501274200bf476635e9f222c5fa05712ecc57af

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp314-cp314t-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp314-cp314t-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 0f2fdc44e604c39c96ecd4a090a7019499be7c11858b8d359d0a7bbf8ad95ffa
MD5 e3486d1e577a033d548f97bd36fb34e4
BLAKE2b-256 f970cb54f92a782080af3e3f0eaf3e3662d2d9dd000488bb5d5d910c4e8c67b8

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp314-cp314t-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp314-cp314t-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 6514b6ecaf489be1c287f94047a194db44899babd53d8916b718f26c5ee8c4ed
MD5 3ade6c3e2bb304fb42b353a1c42233aa
BLAKE2b-256 6b1045063164822e037e0206ae532f05c20cbab45b5585a79d53107b9273d607

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp310-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 6153050f01b98eaf126f54ccb7c5de3490030e86f8757880330ddd70f5c78f3a
MD5 3b07c1b06461480d4ca9e238932146cf
BLAKE2b-256 6e9ea8e5399335e9153047aeaf6de3dd01d7f9842e53e154f43a7de9338e4a66

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp310-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp310-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 a56f6ccf5639163f299339eca10359fd3b5e5fd5494470e9338e2ee1d0e80cfc
MD5 af31b5ec08284ab8d13e163079515d5b
BLAKE2b-256 1de973c8a8ae49a90273e441f466d0cb10bfd3d7bd642f2e1a6b586762b013a7

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp310-abi3-musllinux_1_2_i686.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp310-abi3-musllinux_1_2_i686.whl
Algorithm Hash digest
SHA256 d0372b62f4d8adf3793111ffb7a547d2a28bd97c723777320fad16cda5f08ace
MD5 d7c5c5c51c03c7ebd56b4f776698f627
BLAKE2b-256 301b22f1285b1cd73497e473a8c10b8bc7eb919c08e25b802275f09cc47dca43

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp310-abi3-musllinux_1_2_armv7l.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp310-abi3-musllinux_1_2_armv7l.whl
Algorithm Hash digest
SHA256 59753bafd88bf16cb676b8f538542c5056ba6127810dc6210b3e05737d073083
MD5 c429e0ae2a5a4fe5bda63fca49246888
BLAKE2b-256 cdc31d45d6d73d5bae109c2edf160dfe38f67bd53bfa6e01b085f1302c57ce87

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp310-abi3-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp310-abi3-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 c6350336dd43b1cbb3f77721922639d570a3ff1d258f7d26cebd5ff8d8afdf10
MD5 50dd420f4db0feb327575ea443e56314
BLAKE2b-256 6b2fe2229493f3542c94a39ca8fff831bd1a990607743d2f6d71e7430ba2db30

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp310-abi3-manylinux_2_28_ppc64le.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp310-abi3-manylinux_2_28_ppc64le.whl
Algorithm Hash digest
SHA256 1bdb5a9a57c1cbc0a839b383298dccddbb75f2466eaf0793b13a28092f5df588
MD5 9129e67270587c11eb3b9fc65dfe2021
BLAKE2b-256 a6f914cfc61db861719ecb1b70ff1a974874cd3272f74f19f687a37f91733540

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp310-abi3-manylinux_2_28_armv7l.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp310-abi3-manylinux_2_28_armv7l.whl
Algorithm Hash digest
SHA256 62a1de4b1038bc7d312b19b8d3d5da908d85b572e8a373be3022893430b42950
MD5 7580d33566d025e75a4d26b0a48156da
BLAKE2b-256 6d91fce0601db3fb730a35de6fdbdcc40225c26612dc91aa4708c89b490a323c

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp310-abi3-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp310-abi3-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 f0608b3b8ad5bdaa69cb7f4e5c47714b508c5e97c1ea4a8aec6fe9a223448aed
MD5 86e9a3033e66425fb154dd2d2ea0b80d
BLAKE2b-256 52edce9d599b751a9f039c411449a0575ae99c16ba5f04c72119eb61076e5bb7

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 89fec91d519fe6ec1fd1b86e9e3790cdd3343bd6441210104681f3ef277382a7
MD5 5786fde7a1aff1c2fb56779d5ceeadd3
BLAKE2b-256 8bf765d020a2eb4281e089bdeb0b118f96e6c553b507d23f94c5a1030bd37135

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm Hash digest
SHA256 3f310d9404e15f48d02d7cf1ed3bf7ebac6e7fc829633c0cd35f31ae4d69654f
MD5 3004c3be1083fdc9cc89fa6ebabadeb0
BLAKE2b-256 4b93d0a67132e16c7c22fdb81d0dfc0cd3651d57f1f5aaece491daf94f458784

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 31341822e6983e5785f82e39b76baa7c1588aaafbf44fcbbbf7dcd2e0b64dd34
MD5 6b4e6899d81a37ea87f8370836cb9795
BLAKE2b-256 896840de7b9bde9274047a553af372e6c843dacb54d9095c86720669ee7c6a7d

See more details on using hashes here.

File details

Details for the file llm_runtime_metrics-0.0.15-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for llm_runtime_metrics-0.0.15-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 eb753fcff7f169b9d0bbd924d8e75ad4b7357a2229d0b3f15eb2755b2ebc0c37
MD5 6574e11ce03c6920ab7b652dd80ff19f
BLAKE2b-256 4d34215f2b5c0614f2cc287a4419cfaf3015878e7771b6b05e6dbe002c34c9f2

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page