TensorPackAI (Community Edition)
Silicon & Node-Level LLM Runtime Energy Optimizer for vLLM & NVIDIA Triton
Real-time GPU power telemetry, Joules-per-Token exporter, and silicon frequency management designed specifically for generative foundation model serving.
Official Website • Documentation • Enterprise Edition • Bug Tracker
📖 Overview
Modern Large Language Model (LLM) serving frameworks such as vLLM and NVIDIA Triton Inference Server operate GPUs at maximum core clock frequencies ($f_{\max}$) throughout both prefill (compute-bound) and decode (memory-bound) phases.
Because dynamic CMOS power dissipation scales quadratically with supply voltage and linearly with operating frequency ($P_{\text{dyn}} \propto V^2 f$), sustaining peak clock frequencies during memory-stalled autoregressive decoding wastes immense electrical power without meaningful latency gains.
TensorPackAI Community Edition provides the essential developer and telemetry layer:
- Zero-Overhead C++17 NVML Engine: Direct dynamic loading (
dlopen) oflibnvidia-ml.soto eliminate Python ctypes latency. - Prometheus Metrics Exporter: Real-time metrics including
tensorpack_joules_per_tokenandtensorpack_tokens_per_wattfor Grafana dashboards. - Developer CLI (
tensorpack-tune): Command-line inspection, static clock pinning, and phase-aware profile recommendations. - Static Rule Governor: Threshold-based dynamic clock scaling for quick energy optimization experiments.
⚡ Architecture
+------------------------------------+
| vLLM / Triton Serving Engine |
+------------------------------------+
|
(Token Generation Stream)
v
+------------------------------------+
| TensorPackAI Community Exporter |
| • Real-Time Energy / Token Metric |
| • Tokens per Watt Efficiency |
+------------------------------------+
|
+---------------+---------------+
| |
v v
+------------------------+ +------------------------+
| Prometheus / Grafana | | `tensorpack-tune` CLI |
| (:9100/metrics) | | (Manual Clock Locking) |
+------------------------+ +------------------------+
| |
+---------------+---------------+
|
v
+------------------------------------+
| Zero-Overhead C++17 Fast NVML Core |
+------------------------------------+
|
v
+------------------------------------+
| NVIDIA GPU Silicon (B200/H100/A100)|
+------------------------------------+
🚀 Quickstart
1. Installation
Install via pip directly from the repository:
git clone https://github.com/tensorpackai/tensorpackai-community.git
cd tensorpackai-community
pip install .
Requirements: Linux x86_64 or aarch64, NVIDIA GPU Driver 525+, Python 3.8+.
2. Live GPU Telemetry & Inspection
Inspect real-time GPU power draw, temperatures, operating clocks, and VRAM occupancy:
tensorpack-tune status
Output Example:
TensorPackAI GPU Telemetry (Driver: 550.54.14):
╒═══════╤══════════════════════════════════╤════════╤═════════╤════════╤════════════╤═════════════╤════════════════╤════════╕
│ GPU │ Device Name │ Tier │ Power │ Temp │ SM Clock │ Mem Clock │ VRAM Used/Total│ SM Util│
╞═══════╪══════════════════════════════════╪════════╪═════════╪════════╪════════════╪═════════════╪════════════════╪════════╡
│ 0 │ NVIDIA H100 80GB HBM3 │ H100 │ 285.0 W │ 52 °C │ 1095 MHz │ 2619 MHz │ 40.0 / 80.0 GB │ 75% │
│ 1 │ NVIDIA H100 80GB HBM3 │ H100 │ 290.0 W │ 53 °C │ 1095 MHz │ 2619 MHz │ 40.0 / 80.0 GB │ 75% │
╘═══════╧══════════════════════════════════╧════════╧═════════╧════════╧════════════╧═════════════╧════════════════╧════════╛
3. Recommend Architecture Profiles
Inspect recommended phase-aware operating clocks for your detected accelerator:
tensorpack-tune profile --gpu 0
=======================================================
TensorPackAI Profile for GPU 0: NVIDIA H100 80GB HBM3 (H100)
=======================================================
• Architecture: Hopper
• TDP / Baseline Idle: 700 W / 150 W
• Prefill Boost Clock (f_max): 1980 MHz (Peak GEMM Compute Bound)
• Decode Throttle Clock (f_mem):1095 MHz (Memory-Paced GEMV Bound)
• Idle Power-Gated Clock (f_min):540 MHz (Deep Energy Gating)
• Estimated Decode Dynamic Drop:~72.9% power reduction (P_dyn ∝ V^2 * f)
=======================================================
4. Lock Operating Clocks (CLI Tuning)
Lock GPU core clock to memory-paced decode frequency (e.g., 1095 MHz on H100) to measure power drops:
# Pin GPU 0 core frequency (Requires CAP_SYS_ADMIN or sudo)
sudo tensorpack-tune pin --gpu 0 --clock 1095
# Reset back to default vendor curve
sudo tensorpack-tune reset --gpu all
5. Launch Prometheus Telemetry Exporter
Start the Prometheus exporter service to scrape real-time hardware energy and vLLM token metrics into Grafana:
tensorpack-exporter --port 9100 --interval 0.5 --vllm-url http://localhost:8000/metrics
The metrics endpoint will be available at http://localhost:9100/metrics.
📊 Prometheus Metrics Reference
| Metric Name | Type | Description |
|---|---|---|
tensorpack_gpu_energy_joules_total |
Counter | Cumulative electrical energy consumed by the GPU since exporter start ($\text{Joules}$). |
tensorpack_generation_tokens_total |
Counter | Cumulative generated tokens observed via vLLM /metrics or runtime hooks. |
tensorpack_joules_per_token |
Gauge | Windowed electrical energy consumed per generated token ($\Delta E / \Delta N_{\text{tok}}$). |
tensorpack_cumulative_joules_per_token |
Gauge | Cumulative electrical energy per generated token since start ($E_{\text{total}} / N_{\text{tok,total}}$). |
tensorpack_tokens_per_watt |
Gauge | Windowed throughput efficiency ($\text{Tokens/s} / \text{Watt}$, equivalent to $\text{Tokens/Joule}$). |
tensorpack_gpu_power_watts |
Gauge | Instantaneous GPU electrical power dissipation in Watts. |
tensorpack_gpu_sm_clock_mhz |
Gauge | Streaming Multiprocessor (SM) operating frequency in MHz. |
tensorpack_gpu_mem_clock_mhz |
Gauge | GPU Memory operating frequency in MHz. |
tensorpack_gpu_temperature_celsius |
Gauge | GPU silicon core temperature in Celsius. |
tensorpack_gpu_utilization_ratio |
Gauge | GPU compute utilization (0.0 to 1.0). |
🏢 TensorPackAI Enterprise
While the Community edition provides essential hardware telemetry, Prometheus energy integration, and manual frequency locking, TensorPackAI Enterprise delivers fully autonomous, micro-batch phase-aware runtime co-design:
- Autonomous Phase-Aware Silicon Governor:
- Distinguishes prefill from decode phases at micro-batch execution boundaries without modifying model weights or application code.
- Amortizes driver invocation latencies ($\Delta_{\text{driver}}^{\text{amortized}} < 0.05\text{ ms/token}$) with zero pipeline blocking.
- Eliminates frequency hunting and protects server VRMs from thermo-mechanical stress during continuous batching.
- Cluster-Level Multi-Objective Batch Orchestrator:
- Optimizes request-to-GPU placement across heterogeneous accelerator pools in background execution threads with Zero Dispatch Latency.
- Guarantees P90 Inter-Token Latency (ITL) < 20 ms while reducing net cluster energy by 22.3% – 58.1% (achieving up to 2.73 Tokens/Joule, a 2.39× efficiency gain).
To request an Enterprise trial or benchmark evaluation for your datacenter:
👉 Visit tensorpackai.com or contact oguzhan.sereflisan@gmail.com.
🔍 Search & Discovery Keywords
vllm energy optimization • vllm power management • vllm dvfs • triton inference server energy • llm inference power consumption • joules per token • tokens per watt • gpu energy telemetry • prometheus nvml exporter • nvidia h100 power capping • nvidia a100 clock locking • nvidia b200 power efficiency • sustainable ai inference • green computing llm • cloud gpu cost reduction • prefill decode phase separation • datacenter carbon footprint reduction
📄 License
TensorPackAI Community Edition is licensed under the Apache License, Version 2.0 (SPDX-License-Identifier: Apache-2.0).
Enterprise runtime governors and cluster orchestration engines are proprietary and protected by pending patent applications.
Metadata
Release files for tensorpackai-community 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| tensorpackai_community-0.1.0.tar.gz | 14.2 kB | Without provenance | Details for tensorpackai_community-0.1.0.tar.gz |
Release files / tensorpackai_community-0.1.0.tar.gz
| Download URL | tensorpackai_community-0.1.0.tar.gz |
|---|---|
| Size | 14.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
85af996aa741722c9d72df0f4d0b6d7e37f8631a101e680aece3a6282dd82daa
|
|
BLAKE2b-256 checksum How to use checksums |
b1dbe6a11aef5902ddee3a312bea6504461907fd57ae39f15123ef0853648fae
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.13
|
Security for tensorpackai-community 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.