tfmx 1.1.2

Serve Transformers and Manage GPUs

tfmx

Install

pip install tfmx --upgrade

Commands

Set GPU control state:

gpu_fan -cs a:1

Set GPU power limit:

# M-X GPU-0/1
gpu_pow -pm a:1 && gpu_pow -pl "0:160;1:200"

# M-A GPU-0/1
gpu_pow -pm a:1 && gpu_pow -pl "0,1:160"

Set GPU fan speed:

gpu_fan -cs a:1 && gpu_fan -fs a:100

Set GPU monitor with curve:

# gpu_mon -c "a:30-50/50-65/60-80/75-100;3,7:25-100" -s

Run tei compose and machine:

tei_compose up && sleep 45 && tei_machine run --perf-track

Run tei benchmark:

tei_benchmark run -E "http://localhost:28800" -n 100000

Run qwn compose, machine, and benchmark:

qwn compose up --gpu-configs "0:4b:4bit"
qwn compose up --gpu-layout uniform-awq
qwn machine run -b
qwn client chat "你好，请做个自我介绍"
qwn benchmark run -E "http://localhost:27800" -n 100