mlenergy-data 0.3.2

ML.ENERGY Benchmark data toolkit

The ML.ENERGY Data Toolkit

A Python toolkit for ML.ENERGY datasets: loading raw results, filtering and analyzing runs, fitting models, and building data packages.

We currently have The ML.ENERGY Benchmark v3.0 dataset, which includes LLM and diffusion inference runs on NVIDIA H100 and B200 GPUs. Actual data are stored in Hugging Face Hub: ml-energy/benchmark-v3. This repository contains the toolkit code, not the data itself.

What the toolkit does

• Load and filter benchmark runs with typed, immutable collection classes (LLMRuns, DiffusionRuns).
• Extract bulk data (power timelines, ITL samples, output lengths) as DataFrames.
• Fit models: logistic power/latency curves, ITL latency distributions.
• Build data packages for publishing to Hugging Face Hub.

Installation

pip install mlenergy-data

Dataset access

The benchmark dataset (ml-energy/benchmark-v3) is gated on Hugging Face Hub. Before using the toolkit to load data from HF, you need to:

1. Visit the dataset page and request access (granted automatically).
2. Set the HF_TOKEN environment variable to a Hugging Face access token.

Quick example

from mlenergy_data.records import LLMRuns

runs = LLMRuns.from_hf()

# Find the minimum-energy model on GPQA
best = min(runs.task("gpqa"), key=lambda r: r.energy_per_token_joules)
print(f"{best.nickname}: {best.energy_per_token_joules:.3f} J/tok on {best.gpu_model}")

energies = [r.energy_per_token_joules for r in runs]

Filter, group, and compare across GPU generations and model architectures:

# Compare GPU generations: best energy efficiency per model on GPQA
for gpu, group in runs.task("gpqa").group_by("gpu_model").items():
    best = min(group, key=lambda r: r.energy_per_token_joules)
    print(f"{gpu}: {best.nickname} @ {best.energy_per_token_joules:.3f} J/tok, "
          f"{best.output_throughput_tokens_per_sec:.0f} tok/s")

# MoE, Dense, Hybrid: who's more energy-efficient?
for arch, group in runs.task("gpqa").gpu_model("B200").group_by("architecture").items():
    best = min(group, key=lambda r: r.energy_per_token_joules)
    print(f"{arch}: {best.nickname} @ {best.energy_per_token_joules:.3f} J/tok")

Who uses it

• The ML.ENERGY Leaderboard v3.0: Benchmark results are loaded and compiled into the leaderboard web app data format.
• OpenG2G: Datacenter-grid coordination simulation framework; loads benchmark data and fits models.
• The ML.ENERGY blog: Analysis scripts for blog posts.

Documentation

See the full documentation site for:

• Usage guide: progressive walkthrough from loading data to fitting models.
• API reference: auto-generated from docstrings.

Citation

@inproceedings{mlenergy-neuripsdb25,
    title={The {ML.ENERGY Benchmark}: Toward Automated Inference Energy Measurement and Optimization},
    author={Jae-Won Chung and Jeff J. Ma and Ruofan Wu and Jiachen Liu and Oh Jun Kweon and Yuxuan Xia and Zhiyu Wu and Mosharaf Chowdhury},
    year={2025},
    booktitle={NeurIPS Datasets and Benchmarks},
}