Lutflow

Measure vLLM throughput and allocated token cost—including idle time.

Lutflow reads the metrics of a vLLM server you already run and writes a local JSON report. Missing data stays explicit. No Lutflow account, cluster agent or GPU sensor is required. The Python SDK also tracks provider spend inside your app.

Start with a running vLLM server

Requires Python 3.9+ and uv. Replace the URL with your engine's base URL, not a router. The server must already expose /metrics.

uvx --from 'lutflow>=0.3.2,<0.4' lutflow check --endpoint http://localhost:8000 &&
uvx --from 'lutflow>=0.3.2,<0.4' lutflow measure --endpoint http://localhost:8000 --duration 60 --interval 5 --json measurement.json &&
python3 -m json.tool measurement.json

These commands check the telemetry, observe a 60-second window, then display the report. measure does not generate traffic. If the server is idle, use probe with permission to send requests (below).

Without your USD hourly rate, monetary fields are unavailable; token counts and throughput can still be reported. An idle window has no defined cost per token. Useful tokens mean prompt + generated tokens, not accepted answers.

What the report tells you

Each metric in tier0.measure.v1 carries its value, unit, status and source. Missing values are null, not zero; warnings describe uncertainty and limits. The report stays on your machine. Live commands do not measure GPU utilization, power, energy or VRAM; these fields remain unavailable. Throughput and productive duty cycle measure neither physical GPU utilization nor answer quality.

Commands

check: before spending a measurement window

Measurable means the metrics expose one engine-process identity, the required typed metric families and counter creation markers. check judges one snapshot; it does not run a benchmark or certify GPU readiness or model quality.

uvx --from 'lutflow>=0.3.2,<0.4' lutflow check --endpoint http://localhost:8000 --json check.json

Exit codes: 0 pass, 1 demonstrated failure, 3 unreadable/no verdict, 2 invalid usage. Read the reported reasons before continuing after a failure. Add --model "$MODEL_ID" to require that model's metrics and /v1/models entry. Use --metrics-file saved.prom instead of --endpoint to check a saved snapshot.

measure: observe existing traffic

To include monetary fields, supply a sourced quote and the allocation it covers. For an instance quote, set USD_PER_HOUR, GPUS_IN_INSTANCE and GPU_COUNT from your price source and deployment record; keep the quote's date and source.

uvx --from 'lutflow>=0.3.2,<0.4' lutflow measure --endpoint http://localhost:8000 \
  --duration 60 --interval 5 --json measurement.json \
  --gpu-hourly-rate "$USD_PER_HOUR" --rate-basis instance \
  --gpus-in-basis "$GPUS_IN_INSTANCE" --gpu-count "$GPU_COUNT"

For a per-GPU quote, use --rate-basis per_gpu and omit --gpus-in-basis. No rate or GPU count is guessed. Cost is calculated from your inputs; it is not a reconciled cloud invoice.

A process-identity change, counter reset or token-counter gap stops the window with exit 1 and no report. An unreadable endpoint returns 2. For a URL with a path, give measure the full metrics URL; check instead appends /metrics to its base URL. --model selects a model when the endpoint exposes several.

lutflow measure --from-telemetry FILE reads an existing .jsonl or .jsonl.gz export offline and prints a table by concurrency: read-only, no network, no live window.

probe: generate explicitly confirmed load

uvx --from 'lutflow>=0.3.2,<0.4' lutflow probe --endpoint http://localhost:8000 \
  --model "$MODEL_ID" --concurrency 4 --duration 60 --json probe.json

This example requests four concurrent streams for 60 seconds. probe shows its plan and requires you to type yes before sending synthetic requests. There is no unattended confirmation flag. Use a bare engine URL: path-bearing proxy URLs are not supported for probe. Reports mark the load as induced.

Python SDK: budgets inside your process

Wrap an OpenAI, Anthropic or Gemini client with Client.wrap(provider_client) to track spend and enforce a local budget. Wrapped calls still go to your provider. Install lutflow[openai]>=0.3.2,<0.4 and set OPENAI_API_KEY; the call uses your key. The USD 10 budget below is an example limit you choose.

import openai
from lutflow import BudgetStrategy, Client

client = Client(tenant_id="my-app", budget_usd=10.0, on_budget_exceeded=BudgetStrategy.RAISE_ERROR)
wrapped = client.wrap(openai.OpenAI())
wrapped.chat.completions.create(model="gpt-4o-mini", messages=[{"role": "user", "content": "Hi!"}])
print(client.accumulated_cost_usd)
print(client.remaining_budget_usd)

The default action raises BudgetExceededError. Warning, callback and self-termination strategies are also available. Enforcement affects your Python process; it does not stop cluster workloads or cap your provider account.

What's new in 0.3.x

• 0.3.2: window-mean TPOT from recent vLLM inter-token histograms; a warning identifies the histogram bucket used to interpolate p95.
• 0.3.1: check evaluates telemetry structure before a measurement window.
• 0.3.0: resets, identity changes and token gaps reject unsafe windows.

Limitations

• vLLM only for live endpoint measurement. The check descriptor targets vLLM v0.30.0 metrics; other layouts can fail preflight. Multi-process exporters can omit the process identity and creation markers this check requires.
• Identity must be published. Without it, measure warns and cannot detect every restart whose counters keep rising. Successful completion alone does not establish process continuity.
• Sampling limits resolution. Productive duty cycle reflects scrape intervals. Histogram p95 is interpolated inside a bucket, not an exact tail measurement. TPOT is a per-token-gap mean, not a per-request mean.
• Wrapper prices need review. The dated built-in table has separate input and output rates. Unrecognized model names silently use gpt-4o-mini pricing. Only the price_per_1k_usd override applies one rate per 1k tokens to both input and output. measure uses your supplied rates instead of that table.
• Endpoint access uses your environment. Proxy and certificate variables apply. JSON from check includes the endpoint path, which can identify a namespace or pod; review it before sharing. No report is uploaded to Lutflow.

Installation and extras

For a persistent CLI:

uv tool install 'lutflow>=0.3.2,<0.4'

For Python code, install inside your virtual environment:

python -m pip install 'lutflow>=0.3.2,<0.4'

Provider extras: 'lutflow[openai]>=0.3.2,<0.4', 'lutflow[anthropic]>=0.3.2,<0.4', 'lutflow[gemini]>=0.3.2,<0.4', or 'lutflow[all]>=0.3.2,<0.4'. They add provider SDKs; the vLLM CLI needs no extra. The anthropic and all extras require Python 3.10+; the CLI and other extras support Python 3.9+.

License

Source available under BSL 1.1, licensed by Lutflow EAS. The Change Date is 2030-03-29, when the Change License becomes Apache-2.0. The packaged LICENSE contains the full terms, including commercial-use restrictions.

Links

• Website
• Package and source distributions
• Contact: oscar@lutflow.com