joules

Like time, but for energy. Wrap any command and get the real GPU, CPU and DRAM energy it used, what that cost, its carbon, and how far it sits above the Landauer limit of physics. Benchmark a local model and get joules per token, and how that compares with an API's price.

Illustrative output:

$ joules -- python train.py
joules · python train.py · 2.31 h · exit 0
  GPU 0 NVIDIA H100 80GB HBM3   4.91 MJ    590 W avg  nvml counter
  CPU package 0                  612 kJ     74 W avg  rapl counter
  DRAM (package 0)              98.7 kJ     12 W avg  rapl counter
  ──────────────────────────────────────────────────────────
  total  5.62 MJ = 1.56 kWh
  cost 0.156 at 0.1/kWh · CO₂ 577 g at 370 g/kWh

$ joules bench ollama:llama3.1:8b --api-price 0.20
  per token      3.74 J   ·   1.04 kWh per 1M tokens
  electricity    0.104 per 1M output tokens at 0.1/kWh
  vs API         0.2 per 1M → electricity is 1.9× cheaper than the API (hardware cost not included)

No dependencies: only the Python standard library and your GPU driver.

On NVIDIA GPUs every measurement also reads the card's clocks, temperature, power limit, busy and memory-controller percentages and the driver's throttle reasons, and says in words what they mean:

  GPU 0 NVIDIA H100 80GB HBM3   4.91 MJ    590 W avg  nvml counter
    1,412 MHz (71% of max) · 94% busy · memory 38% · 83 °C max · limit 700 W · compute-bound · throttled 62% of the time (thermal)

Throttled means below the maximum clock for a reason other than having nothing to do: the power cap, heat, a hardware slowdown or a power brake. Compute-bound or memory-bound is a coarse reading from the two busy percentages, and is labelled as such. --track sends these as numbers, so the Tracking page shows which runs were throttled.

Install

macOS and Linux:

curl -LsSf https://wattops.dev/install.sh | sh && export PATH="$HOME/.local/bin:$PATH"
joules doctor                                # what can be measured here, with a live reading from each meter

Windows 10 or 11, in PowerShell (no admin rights):

irm https://wattops.dev/install.ps1 | iex
joules doctor

The installers need only Python 3.9+ (no sudo, no pip) and update joules when run again. Or use pipx: pipx install landauer-gap (update: pipx upgrade landauer-gap).

Commands

Useful options: --json / --out FILE (machine-readable result), --subtract-idle 5 (measure the idle machine first and also report energy above idle), --gpus 0,1, --price, --grid fr or --ci 55, --pue 1.2.

Tracking your runs over time

joules login lgk_…                              # once: saves your key (Settings → API keys shows this line)
joules run --track llama-ft --params 8 --tokens 2 -- python train.py
joules --track nightly -- ./train.sh            # short form

Each tracked run appears on the Tracking page within seconds: Landauer gap over time, energy per run and totals for the project. joules sends the energy, time, cost, CO₂, hardware, operations and gap, and a short label: the program and script name (python train.py) or your --label. It never sends arguments, inline code, outputs or file contents. A tracking problem is reported but never changes the command's exit code, so it is safe in CI.

Comparing two ways of doing the same work

joules compare --names new,old --runs 7 --subtract-idle 10 --markdown results.md \
  "./bench --runtime new" "./bench --runtime old"

Runs a warm-up round, then the commands interleaved (A B A B, or --order abba), and reports per command the median energy, time, average power and energy × time with the spread of the middle half of the runs, the GPU's clock, throttling and temperature, and against the baseline the change with a Mann-Whitney p-value. It warns when one side was throttled more, ran at a different clock or started hotter, and --limits 100,85,70,55 (sudo) repeats the comparison at fixed GPU power limits so a gain cannot come from the card boosting differently. The full method is in docs/BENCHMARKING.md.

Phases of your own program

Inside a Python program:

import joules

with joules.Session() as s:
    with s.phase("incremental update") as p:
        runtime.update(batch)
    print(p.energy_j, p.duration_s, p.gpu)
    print(s.summary())          # per phase name: count, median energy and time, quartiles

From any language, with joules agent running on the machine:

curl -s -X POST localhost:9877/phase/start -H 'content-type: application/json' -d '{"name": "incremental update"}'
curl -s -X POST localhost:9877/phase/end   -H 'content-type: application/json' -d '{"id": "p1"}'
curl -s localhost:9877/phases               # medians per phase name

Each mark reads every meter at that moment, so a phase's energy is exact to the counter's resolution. Phases can nest and overlap, and the end of each one also says what the GPU did during it (clock, throttling, temperature at start and end). The phase endpoints refuse requests from web browsers.

Energy check on every pull request

joules ci runs a command on your checkout and on the base branch (a temporary git worktree), alternating runs so drift hits both sides, and compares the medians. It writes a Markdown summary, can post one pull request comment that it updates on each push, and fails when energy grows by more than --budget percent. A change smaller than the run-to-run spread is reported as noise, never as a failure.

joules ci --base main --runs 3 --budget 10 -- pytest -q

In GitHub Actions use the PR energy check action. On hosted runners, which have no energy counters, energy is estimated from CPU time and the comment says so; on a self-hosted runner with a GPU or RAPL it is measured. joules run --estimate uses the same estimate when a machine has no counters.

Energy receipts for AI responses

joules proxy sits in front of a model server and measures the hardware while each request runs. Point your client at the proxy instead of the server; nothing else changes.

joules proxy --upstream http://localhost:11434          # Ollama
joules proxy --upstream http://localhost:8000 --subtract-idle 5   # vLLM, measured above idle

$ curl -si localhost:8787/v1/chat/completions -d '{"model":"llama3.1:8b","messages":[…]}'
X-Energy-Joules: 41.2
X-Energy-Output-Tokens: 212
X-Energy-Joules-Per-Token: 0.194
X-Energy-CO2-Grams: 0.00423
X-Energy-Receipt: /joules/receipts/r-000017

• Streaming responses (SSE or Ollama's NDJSON) pass through untouched and carry X-Energy-Receipt; the receipt at that address is complete when the stream ends.
• Tokens come from the response (usage, or Ollama's eval_count). Without them, streamed pieces are counted and the receipt says tokens_exact: false.
• Meters measure whole devices, so in each sampling interval the energy is shared equally between the requests running at that moment. With --subtract-idle receipts also carry energy above idle.
• /joules/receipts lists the newest receipts, /joules/metrics serves Prometheus metrics, and /joules/health says what is measured.
• --track PROJECT sends one summary per model every 5 minutes to your Tracking page: never prompts or outputs.
• It listens on 127.0.0.1 by default. --host 0.0.0.0 exposes it, and your model server, to your network.

Energy per job on GPU clusters

joules agent runs on every node of a cluster, measures it every second and splits each device's energy between the jobs on it: NVIDIA GPUs by the GPU memory each process holds, AMD and Intel GPUs by GPU time, CPUs and DRAM by CPU time. Jobs are Slurm jobs, Kubernetes pods, containers or users' programs. GPUs a job holds but barely uses are reported as idle GPU energy.

sudo joules agent                         # http://127.0.0.1:9877/jobs and /metrics (Prometheus)
sudo joules agent --track cluster         # each finished job on your Tracking page

On the Max plan, --track also puts every node on the Live page (https://wattops.dev/#/live): its power, cost per hour, idle GPUs and the jobs running now, updated every 5 seconds (--live-every, 0 turns it off), and every GPU on its own: power, how busy, clock, temperature, power limit and whether it is throttled and why, so a fleet of a hundred GPUs shows which ones are idle, hot or held back. Heartbeats do not count against your API calls. On a Mac the agent measures the whole machine (joules agent --track fleet), so you can watch it live too. A sleeping machine sends nothing, so Live shows a gap until it wakes; --keep-awake keeps it from sleeping while the agent runs (closing a laptop's lid still puts it to sleep).

A Kubernetes DaemonSet, a systemd unit for Slurm nodes, Prometheus scrape and alert rules and a Grafana dashboard are available: ask us.

The leaderboard

joules login lgk_…                           # once; free key: wattops.dev → Settings → API keys
joules bench ollama:llama3.1:8b --subtract-idle 5 --share

The leaderboard ranks models and machines by measured joules per output token, using the median of everyone's runs. joules sends only the model name, hardware name and count, energy per token, tokens per second, average power, output tokens, whether idle power was subtracted, which meters were used and the joules version. It never sends prompts, outputs, host names or paths. Simulated runs are refused, and you can remove your own entries on the Leaderboard page.

Estimate vs measured, and calibration

Tell joules the size of the training job and it compares the measurement with the WattOps estimate, then works out what your hardware really achieved:

$ joules run --params 1 --tokens 2 -- python finetune.py        # 1B model, 2B tokens, one H100
  estimate vs measured  17 MJ estimated (40% MFU, 80% draw) → 14.8 MJ measured (-13%)
  your hardware ran at 42.5% effective utilisation and 74% of rated power
  open in the calculator with these numbers: https://wattops.dev/?s=…

The link opens the website calculator with your measured utilisation and power draw, so every future estimate starts from your real hardware instead of a textbook default. If the job you describe could not fit in the measured time, joules says so instead of reporting nonsense.

What it measures, and how

Where there is no counter (cloud VMs, CI runners, Windows CPUs), joules estimates energy from CPU time and labels every such result ESTIMATE; estimates are never added to the leaderboard.

Meters cover whole devices, so anything else running on the same GPU or CPU is included. Use --subtract-idle, and keep the machine otherwise quiet, for the cleanest numbers. Power supply losses, fans, networking and cooling are not measured; --pue adds a facility overhead.

joules finds the NVIDIA driver's library where the loader looks and also in the folders drivers are known to use (Google Colab and Kaggle, Debian and Ubuntu, RHEL, NVIDIA's container toolkit, WSL 2, NixOS); if it still cannot, joules doctor says so and JOULES_NVML_LIB=/path/to/libnvidia-ml.so.1 names it.

CUDA_VISIBLE_DEVICES is respected, with indices, GPU UUIDs (as Kubernetes and Slurm set them) or MIG slices (MIG measures the whole GPU). On AMD, HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES and --gpus pick cards. Reading RAPL on recent Linux kernels needs root, or sudo chmod a+r /sys/class/powercap/intel-rapl:*/energy_uj.

Tests

PYTHONPATH=src python3 -m unittest discover -s tests -v

The real sysfs layouts are rebuilt in a temporary directory (including counter wraparound), the NVML binding runs against a compiled stand-in for libnvidia-ml.so.1, and model servers are local HTTP stand-ins. JOULES_SIMULATE="gpu:H100:650,cpu:package 0:80" runs everything on constant simulated power for demos; results are always labelled SIMULATED.