Lutflow — AI Cost Intelligence for LLM inference

Lutflow measures the allocated cost of a vLLM deployment you already run, including idle GPU time. It reports cost per useful token, throughput and the source or limitation of each number. The SDK also tracks and enforces provider spend inside your Python process (OpenAI · Anthropic · Gemini).

Live vLLM measurement requires Lutflow 0.2.1 or newer. Check the installed version before using measure --endpoint: older releases support only the offline --from-telemetry path. For a published release, install from PyPI as shown below. For an unreleased first-operator candidate, use the one audited wheel and checksum in the supplied TIER0_FIRST_OPERATOR.md guide.

Tier 0 is local: no Lutflow account, hosted backend, Kubernetes agent or GPU sensor is needed. To measure a live deployment, you need vLLM metrics reachable from the machine running this command. A sourced USD hourly rate is needed only for monetary fields.

First vLLM measurement (version 0.3.0+)

Use Python 3.9 or newer (CI tests 3.9–3.15). Before installing, check that your vLLM server exposes token counters (replace the URL if it is not on this machine):

curl -fsS http://localhost:8000/metrics | grep -E 'vllm:(prompt|generation)_tokens_total'

The quickest way to use the lutflow command is to run it with uv or pipx: each keeps lutflow in its own isolated environment, so nothing you already have installed changes.

# uv: run it without installing anything permanently
uvx --from 'lutflow>=0.3.0,<0.4' lutflow --version

# uv: install the command for good
uv tool install 'lutflow>=0.3.0,<0.4'

# pipx: install the command for good
pipx install 'lutflow>=0.3.0,<0.4'

The version range makes the install fail instead of quietly picking an older release that lacks the 0.3.0 window checks. With uvx, write every command as uvx --from 'lutflow>=0.3.0,<0.4' lutflow … (for example … lutflow measure …); after uv tool install or pipx install, the command is just lutflow.

To use lutflow from your own Python code, install it in a virtual environment:

python3 --version
python3 -m venv .venv
source .venv/bin/activate
python -m pip install 'lutflow>=0.3.0,<0.4'
lutflow --version

In a uv project, uv add 'lutflow>=0.3.0,<0.4' does the same.

Then measure:

lutflow measure --endpoint http://localhost:8000 \
  --duration 300 --interval 10 \
  --json measurement.json

With no real traffic, the cost per useful token is undefined; use lutflow probe only if you are allowed to generate requests on that server. It shows a load plan and asks for yes before it starts. The report and JSON stay on your machine. Without a sourced USD hourly rate, monetary fields are unavailable.

Capabilities in version 0.2.1+

Budget enforcement acts only on the current Python process (raise / warn / callback / SIGKILL of os.getpid()) — it is not cluster-level workload termination.

The connected Lutflow platform — fleet-wide cost intelligence — is in private development.

In-process budget enforcement

Wrap your provider client; Lutflow meters every call and enforces the limit locally. This runs entirely in your process — no account, no network.

from lutflow import Client, BudgetStrategy
import openai

client = Client(
    tenant_id="acme",
    budget_usd=10.00,
    on_budget_exceeded=BudgetStrategy.RAISE_ERROR,
)
wrapped = client.wrap(openai.OpenAI())
response = wrapped.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
)

print(f"Spent:     ${client.accumulated_cost_usd:.4f}")
print(f"Remaining: ${client.remaining_budget_usd:.4f}")

When the budget is exceeded, the configured strategy fires in this process:

A context-manager form is also available:

from lutflow import budget_session

with budget_session(budget_usd=0.50, tenant_id="acme") as session:
    wrapped = session.wrap(openai.OpenAI())
    ...

GPU-time pricing (self-hosted models)

from lutflow import Client, PricingMode

client = Client(
    tenant_id="acme",
    budget_usd=5.00,
    pricing_mode=PricingMode.GPU_TIME,
    gpu_type="nvidia-l4",
)
client.start_gpu_timer()
# ... run inference ...
cost = client.stop_gpu_timer()

lutflow measure --endpoint — measure a vLLM server you already run

Available from version 0.2.1. Check lutflow --version after installation; 0.2.0 supports only the offline --from-telemetry path.

Point it at a vLLM server that is already serving. It reads that server's own Prometheus metrics across a time window and reports what your tokens cost. No Kubernetes, no Lutflow account, no GPU sensor, no agent to install.

lutflow measure --endpoint "$METRICS_URL" \
  --duration 300 \
  --gpu-hourly-rate "$GPU_RATE_USD_PER_HOUR" --rate-basis "$RATE_BASIS" \
  --gpus-in-basis "$GPUS_IN_BASIS" --gpu-count "$GPU_COUNT" \
  --json measurement.json

Set these variables from your own sourced USD/hour quote and deployment record; record the quote's source and effective date. RATE_BASIS is instance or per_gpu; GPUS_IN_BASIS is 1 for a per-GPU quote. Without a verifiable USD rate, omit all four pricing options and measure without monetary fields. No currency conversion is performed. --rate-basis has no default. Some cloud offers quote an entire multi-GPU instance. Reading that price as a per-GPU rate overstates cost by the number of GPUs included, so Lutflow asks which basis the quote uses.

Add your API price to get a comparison and a break-even point:

lutflow measure --endpoint "$METRICS_URL" \
  --gpu-hourly-rate "$GPU_RATE_USD_PER_HOUR" --rate-basis "$RATE_BASIS" \
  --gpus-in-basis "$GPUS_IN_BASIS" --gpu-count "$GPU_COUNT" \
  --api-provider "$API_PROVIDER" --api-model "$API_MODEL" \
  --api-input-rate "$API_INPUT_USD_PER_MILLION" \
  --api-output-rate "$API_OUTPUT_USD_PER_MILLION" \
  --api-price-effective-at "$API_PRICE_EFFECTIVE_AT" \
  --api-price-source "$API_PRICE_SOURCE"

Every field is required for a comparison, including where the price came from and when it was in force. There is no built-in price table: prices move, and a substituted rate produces a comparison against a bill you never received.

The output has two halves, in this order:

• Allocated economics — the whole window, idle seconds included. This is what the deployment costs. A reserved GPU bills while it waits.
• Active efficiency — only the seconds that produced useful tokens. This is how well the deployment runs when it is running.

They coincide at 100% duty cycle and diverge everywhere else. The gap between them is what you spend on idle capacity.

Useful tokens are prompt + generation. Anything a GPU sensor would have provided reports unavailable, never 0 — this slice reads the inference engine only, so power, energy, VRAM and GPU utilisation are all unavailable. vLLM does not publish GPU utilisation, and KV-cache occupancy is a different quantity that is not substituted for it.

The CLI fetches metrics from your vLLM endpoint. The report and JSON export stay on your machine; the JSON contains no endpoint, hostname, secret or prompt text, and nothing is sent to Lutflow. probe sends synthetic requests to the endpoint only after you confirm its load plan.

It does not guess a rate, substitute a similar model's price, assume you bought one GPU, or add two models' tokens together. Missing prices stay unavailable; incomplete cost inputs and ambiguous model selection produce actionable errors.

Currently supported: vLLM. TGI and Ollama are not implemented, and asking for them says so rather than silently using vLLM's metric names.

Limits

• A window comes from one engine process, with continuous counters. From 0.3.0, a token counter or latency histogram that goes backwards, a token counter that disappears from some scrapes, or a change in the engine's process identity ends the measurement with exit 1 and no report. The tokens around a restart were never observed, so no total is published.
• Restart detection depends on the engine publishing its identity. If the engine publishes neither process_start_time_seconds nor the _created series of its token counters, a restart whose counters keep rising can go unnoticed. The report then states engine_process_identity_unpublished_restart_with_rising_counters_not_detectable in warnings[] and is not rejected. The absence of a reset error does not prove that the window came from a single continuous process.
• TPOT is unavailable on recent vLLM. 0.3.0 reads vllm:time_per_output_token_seconds, which recent vLLM (v0.30.0, for example) no longer publishes. tpot_ms is then reported unavailable, never estimated; TTFT, end-to-end latency and tokens are unaffected. See Known limitations.
• Wrapper costs use a dated price table. Model names outside it are mapped to older entries without a warning, so the rate applied may not match the model you call, and a cost can be off by an order of magnitude. Pass price_per_1k_usd to Client when you rely on them. lutflow measure is not affected. See Known limitations.
• The p95 resolves only to a histogram bucket. It is interpolated inside the engine's bucket that contains it: with vLLM's default end-to-end buckets, a window whose requests all take 2.5–5 s reports 4875 ms whatever the real tail. Read it as "within that bucket". See Known limitations.

lutflow probe — measure a vLLM server with no traffic yet

Available from version 0.2.1, with explicit confirmation before load starts.

--endpoint measures traffic your server is already handling. probe is for the opposite case: a server that is up but idle, with nothing yet to observe. It generates its own load against the server, then measures that load with the exact same path --endpoint uses.

lutflow probe --endpoint "$METRICS_URL" --model "$MODEL_ID" \
  --concurrency 8 --duration 90 \
  --json measurement.json

Because it generates traffic on a server you may not own alone, four guards apply and none of them can be turned off:

• Opt-in. probe is a separate command; measure --endpoint never generates load on its own.
• Declared, not disguised. Every probe document is marked measurement_mode: "probe" and load_origin: "induced", and the terminal output leads with a banner saying the traffic was generated, not observed.
• Synthetic prompts, hard caps. The load uses a fixed, packaged prompt pool (never your data) under concurrency, duration and total-request ceilings that are compiled into the package. No flag, environment variable, or config file can raise them.
• Confirmation, every time. probe prints exactly what it is about to do and requires you to type yes. There is no --force or --yes flag.

If --max-requests finishes the load before the observation window closes, the rest of the window is idle time the probe itself created, and the document's warnings[] (and the terminal) say so rather than let a collapsed duty cycle look like something the endpoint did.

lutflow measure --from-telemetry — offline pass over an existing export

The original offline path, unchanged: point it at a GPU telemetry export for a per-concurrency breakdown of cost and energy per token. Read-only, no network.

lutflow measure --from-telemetry telemetry.jsonl --price-per-hr "$GPU_RATE_USD_PER_HOUR"

Its $/1M useful column is an active figure — cost over each concurrency bucket's productive intervals. A concurrency bucket is a subset of a window by construction, so it carries no idle share of its own; use --endpoint when you need the allocated cost of a real window.

Installation

pip install 'lutflow>=0.3.0,<0.4'            # Core + CLI
pip install 'lutflow[openai]>=0.3.0,<0.4'    # + OpenAI wrapper
pip install 'lutflow[anthropic]>=0.3.0,<0.4' # + Anthropic wrapper
pip install 'lutflow[gemini]>=0.3.0,<0.4'    # + Google Gemini wrapper
pip install 'lutflow[all]>=0.3.0,<0.4'       # All providers

Provider wrappers support OpenAI, Anthropic and Google Gemini. Live endpoint measurement supports vLLM only. In-process GPU-time pricing can use an operator-supplied rate for other self-hosted inference processes; it does not scrape their endpoint metrics.

Links

• Website: lutflow.dev
• PyPI: pypi.org/project/lutflow
• Support: oscarmatiasg@lutflow.com

License

Source-available under the Business Source License 1.1 (BSL 1.1). Internal use, development, testing, evaluation, and non-commercial use are expressly permitted. Converts to Apache-2.0 on 2030-03-29 (the Change Date).

See LICENSE for the full terms, including the commercial-use exclusion, which is not summarized here. For commercial licensing: licensing@lutflow.com