Lutflow

Measure vLLM throughput and allocated token cost—including idle time.

Lutflow reads the metrics of a vLLM server you already run and writes a local JSON report. Missing data stays explicit. No Lutflow account, cluster agent or GPU sensor is required. The Python SDK also tracks provider spend inside your app.

Start with a running vLLM server

Requires Python 3.9+ and uv. Replace the URL with your engine's base URL, not a router. The server must already expose /metrics.

uvx --from 'lutflow>=0.3.3,<0.4' lutflow check --endpoint http://localhost:8000 &&
uvx --from 'lutflow>=0.3.3,<0.4' lutflow measure --endpoint http://localhost:8000 --duration 60 --interval 5 --json measurement.json &&
python3 -m json.tool measurement.json

These commands check the telemetry, observe a 60-second window, then display the report. measure does not generate traffic. If the server is idle, use probe with permission to send requests (below).

Without your USD hourly rate, monetary fields are unavailable; token counts and throughput can still be reported. An idle window has no defined cost per token. Useful tokens mean prompt + generated tokens, not accepted answers.

What the report tells you

Each metric in tier0.measure.v1 carries its value, unit, status and source. Missing values are null, not zero; warnings describe uncertainty and limits. The report stays on your machine. Live commands do not measure GPU utilization, power, energy or VRAM; these fields remain unavailable. Throughput and productive duty cycle measure neither physical GPU utilization nor answer quality.

Commands

check: before spending a measurement window

Measurable means the metrics expose one engine-process identity, the required typed metric families and counter creation markers. check judges one snapshot; it does not run a benchmark or certify GPU readiness or model quality.

uvx --from 'lutflow>=0.3.3,<0.4' lutflow check --endpoint http://localhost:8000 --json check.json

Exit codes: 0 pass, 1 demonstrated failure, 3 unreadable/no verdict, 2 invalid usage. Read the reported reasons before continuing after a failure. Add --model "$MODEL_ID" to require that model's metrics and /v1/models entry. Use --metrics-file saved.prom instead of --endpoint to check a saved snapshot.

measure: observe existing traffic

To include monetary fields, supply a sourced quote and the allocation it covers. For an instance quote, set USD_PER_HOUR, GPUS_IN_INSTANCE and GPU_COUNT from your price source and deployment record; keep the quote's date and source.

uvx --from 'lutflow>=0.3.3,<0.4' lutflow measure --endpoint http://localhost:8000 \
  --duration 60 --interval 5 --json measurement.json \
  --gpu-hourly-rate "$USD_PER_HOUR" --rate-basis instance \
  --gpus-in-basis "$GPUS_IN_INSTANCE" --gpu-count "$GPU_COUNT"

For a per-GPU quote, use --rate-basis per_gpu and omit --gpus-in-basis. No rate or GPU count is guessed. Cost is calculated from your inputs; it is not a reconciled cloud invoice.

A process-identity change, counter reset or token-counter gap stops the window with exit 1 and no report. An unreadable endpoint returns 2. For a URL with a path, give measure the full metrics URL; check instead appends /metrics to its base URL. --model selects a model when the endpoint exposes several.

lutflow measure --from-telemetry FILE reads an existing .jsonl or .jsonl.gz export offline and prints a table by concurrency: read-only, no network, no live window.

probe: generate explicitly confirmed load

uvx --from 'lutflow>=0.3.3,<0.4' lutflow probe --endpoint http://localhost:8000 \
  --model "$MODEL_ID" --concurrency 4 --duration 60 --json probe.json

This example requests four concurrent streams for 60 seconds. probe shows its plan and requires you to type yes before sending synthetic requests. There is no unattended confirmation flag. Use a bare engine URL: path-bearing proxy URLs are not supported for probe. Reports mark the load as induced.

Python SDK: budgets inside your process

Wrap an OpenAI, Anthropic or Gemini client with Client.wrap(provider_client) to track spend and enforce a local budget. Wrapped calls still go to your provider. Install lutflow[openai]>=0.3.3,<0.4 and set OPENAI_API_KEY; the call uses your key. lutflow ships no price table: declare the price you pay for each exact model id, where you read it and the date it applied. The figures below are placeholders.

import openai
from lutflow import BudgetStrategy, Client, TokenPrice

client = Client(
    tenant_id="my-app",
    budget_usd=10.0,  # an example limit you choose
    on_budget_exceeded=BudgetStrategy.RAISE_ERROR,
    token_prices={
        "gpt-4o-mini": TokenPrice(
            input_usd_per_million=0.15,  # replace with the price you pay
            output_usd_per_million=0.60,
            source="<the price page you read>",
            effective_at="2026-10-01",
        ),
    },
)
wrapped = client.wrap(openai.OpenAI())
wrapped.chat.completions.create(model="gpt-4o-mini", messages=[{"role": "user", "content": "Hi!"}])
print(client.accumulated_cost_usd)
print(client.remaining_budget_usd)

The budget is checked before each wrapped call. The call that crosses it returns its response and its spend is recorded; the next wrapped call raises BudgetExceededError before it reaches the provider (WARN_ONLY lets it through, CALLBACK is notified). Spend already made is not reverted: this is a local guard, not a cap on your provider account, and it does not stop cluster workloads. SELF_KILL no longer terminates the process from a wrapped call.

Metered: chat.completions.create and embeddings.create (OpenAI), messages.create (Anthropic), generate_content and chat send_message (Gemini), synchronous and without stream=True, priced from the usage the provider returns. Other methods, streaming and async clients raise UnsupportedMethodError without calling the provider. A model without a declared price raises PriceUnavailableError. A response without usage is returned, and later wrapped calls raise SpendUnverifiableError; so do the calls after a provider call that raised, because that request may have run.

What's new in 0.3.x

• 0.3.3: wrapper integrity: no built-in prices or substituted rates, the response that crosses the budget is kept and later calls stop, and methods the wrappers do not meter are refused.
• 0.3.2: window-mean TPOT from recent vLLM inter-token histograms; a warning identifies the histogram bucket used to interpolate p95.
• 0.3.1: check evaluates telemetry structure before a measurement window.
• 0.3.0: resets, identity changes and token gaps reject unsafe windows.

Limitations

• vLLM only for live endpoint measurement. The check descriptor targets vLLM v0.30.0 metrics; other layouts can fail preflight. Multi-process exporters can omit the process identity and creation markers this check requires.
• Identity must be published. Without it, measure warns and cannot detect every restart whose counters keep rising. Successful completion alone does not establish process continuity.
• Sampling limits resolution. Productive duty cycle reflects scrape intervals. Histogram p95 is interpolated inside a bucket, not an exact tail measurement. TPOT is a per-token-gap mean, not a per-request mean.
• Wrapper costs come from your prices. A cost is the provider-reported tokens times the price you declare; it is not reconciled with an invoice. OpenAI and Gemini cached input tokens are priced at the full input rate; Anthropic cache tokens leave the call unmetered. The deprecated price_per_1k_usd applies one rate to input and output.
• Endpoint access uses your environment. Proxy and certificate variables apply. JSON from check includes the endpoint path, which can identify a namespace or pod; review it before sharing. No report is uploaded to Lutflow.

Installation and extras

For a persistent CLI:

uv tool install 'lutflow>=0.3.3,<0.4'

For Python code, install inside your virtual environment:

python -m pip install 'lutflow>=0.3.3,<0.4'

Provider extras: 'lutflow[openai]>=0.3.3,<0.4', 'lutflow[anthropic]>=0.3.3,<0.4', 'lutflow[gemini]>=0.3.3,<0.4', or 'lutflow[all]>=0.3.3,<0.4'. They add provider SDKs; the vLLM CLI needs no extra. The anthropic and all extras require Python 3.10+; the CLI and other extras support Python 3.9+.

License

Source available under BSL 1.1, licensed by Lutflow EAS. The Change Date is 2030-03-29, when the Change License becomes Apache-2.0. The packaged LICENSE contains the full terms, including commercial-use restrictions.

Links

• Website
• Package and source distributions
• Contact: oscar@lutflow.com