Repository of AutoRound: Advanced Weight-Only Quantization Algorithm for LLMs
Project description
AutoRound
Advanced Weight-Only Quantization Algorithm for LLMs
AutoRound is an advanced weight-only quantization algorithm for low-bits LLM inference. It's tailored for a wide range of models and consistently delivers noticeable improvements, often significantly outperforming SignRound with the cost of more tuning time for quantization.
Prerequisites
- Python 3.9 or higher
Installation
Build from Source
pip install -r requirements.txt
python setup.py install
Usage
On CPU/GPU
from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_round import AutoRound
model_name = "meta-llama/Llama-2-7b-hf"
model = AutoModelForCausalLM.from_pretrained(
model_name, low_cpu_mem_usage=True, torch_dtype="auto", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
bits, group_size, scheme = 4, 128, "asym"
autoround = AutoRound(model, tokenizer, bits=bits, group_size=group_size, scheme=scheme)
autoround.quantize()
# Intel CPU Inference, For now only support llama, mistral and gpt-j.
# then follow ITREX(https://github.com/intel/intel-extension-for-transformers/tree/main/intel_extension_for_transformers/llm/runtime/neural_speed) to load the model and do inference
# currently please install neural-speed (https://github.com/intel/neural-speed) from source
output_dir = "./tmp_autoround"
autoround.export(output_dir)
from intel_extension_for_transformers.transformers import AutoModelForCausalLM, WeightOnlyQuantConfig
woq_config = WeightOnlyQuantConfig(group_size=group_size, scheme=scheme, use_autoround=True) ##only supports 4 bits currently
prompt = "Once upon a time, a little girl"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
inputs = tokenizer(prompt, return_tensors="pt").input_ids
model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=woq_config, trust_remote_code=True,device="cpu")
outputs = model.generate(inputs, max_new_tokens=30)
Tuning on Intel Gaudi2
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "meta-llama/Llama-2-7b-hf"
model = AutoModelForCausalLM.from_pretrained(
model_name, low_cpu_mem_usage=True, torch_dtype="auto", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
bits, group_size, scheme = 4, 128, "asym"
# need to load model first, then import
from auto_round import AutoRound
autoround = AutoRound(model, tokenizer, bits=bits, group_size=group_size, scheme=scheme,
device="hpu", scale_dtype="bf16", amp=False)
autoround.quantize()
Detailed Hyperparameters
-
model
: The PyTorch model to be quantized. -
tokenizer
: An optional tokenizer for processing input data. If none is provided, a dataloader must be supplied. -
bits (int)
: Number of bits for quantization (default is 4). -
group_size (int)
: Size of the quantization group (default is 128). -
scheme (str)
: The quantization scheme (sym/asym) to be used (default is "asym"). -
use_quant_input (bool)
: Whether to use the output of the previous quantized block as the input for the current block (default is True). -
enable_minmax_tuning (bool)
: Whether to enable weight min-max tuning (default is True). -
iters (int)
: Number of tuning iterations (default is 200). -
lr (float)
: The learning rate for rounding value (default is None, it will be set to 1.0/iters automatically). -
minmax_lr (float)
: The learning rate for min-max tuning (default is None, it will be set to lr automatically). -
n_samples (int)
: Number of samples for tuning (default is 512). -
seqlen (int)
: Data length of the sequence for tuning (default is 2048). -
bs (int)
: Batch size for training (default is 8). -
amp (bool)
: Whether to use automatic mixed precision (default is True). -
n_blocks (int)
: Packing several blocks as one for tuning together (default is 1). -
gradient_accumulate_steps (int)
: Number of gradient accumulation steps (default is 1). -
low_gpu_mem_usage (bool)
: Whether to save GPU memory at the cost of a little tuning time (default is True). -
dataset_name (str)
: The default dataset name for tuning (default is "NeelNanda/pile-10k"). -
dataset_split (str)
: The split of the dataset to be used for tuning (default is "train"). -
dataloader
: The dataloader for tuning data. -
weight_config (dict)
: Configuration for weight quantization (default is an empty dictionary), mainly for mixed bits or mixed precision. -
device
: The device to be used for tuning (default is "cuda:0").
Validated Models
For wikitext2/ptb-new/c4-new ppl, we follow the code of gptq and set the sequence length to 2048. For lm-eval wikitext ppl, we adopt lm-eval. The quantization configure is W4G128.
Model | Method | Acc AVG. | MMLU | Lamb. | Hella. | Wino. | Piqa | Truth. | Open. | Boolq | RTE | ARC-e | ARC-c. | wikitext2 ppl | ptb_new ppl | c4_new ppl | lm_eval wikitext ppl |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Intel/neural-chat-7b-v3 | FP16 | 67.92 | 61.13 | 73.03 | 66.39 | 76.40 | 81.01 | 47.37 | 38.8 | 86.97 | 75.81 | 82.66 | 57.51 | 6.00 | 48.96 | 9.65 | - |
Ours | 66.90 | 60.56 | 72.19 | 65.28 | 75.37 | 81.18 | 46.76 | 36.0 | 86.91 | 73.29 | 81.73 | 56.66 | 6.21 | 59.78 | 10.01 | - | |
Ours iters=1K,use_quant_input=False, minmax_lr=0.002 | 67.70 | 60.57 | 73.74 | 65.62 | 77.43 | 80.85 | 47.61 | 36.8 | 86.94 | 75.09 | 82.66 | 57.34 | 6.17 | 59.12 | 9.83 | - | |
mistralai/Mixtral-8x7B-v0.1 | BF16 | 67.16 | 69.83 | 78.44 | 64.89 | 76.40 | 82.43 | 34.15 | 35.40 | 84.98 | 71.12 | 84.22 | 56.91 | 3.84 | 19.22 | 7.41 | - |
Ours | 65.98 | 68.90 | 78.11 | 64.31 | 74.27 | 82.10 | 30.97 | 34.20 | 84.57 | 67.87 | 83.96 | 56.57 | 4.08 | 354 | 7.56 | - | |
Ours iters=1K,use_quant_input=False | 66.78 | 68.68 | 78.61 | 64.40 | 76.56 | 81.99 | 32.56 | 34.80 | 85.96 | 70.76 | 83.96 | 56.31 | 3.99 | 17.65 | 7.52 | - | |
microsoft/phi-2 | FP16 | 61.80 | 56.40 | 62.78 | 55.83 | 75.77 | 78.67 | 31.21 | 40.40 | 83.36 | 62.45 | 80.05 | 52.90 | 9.71 | 18.16 | 14.12 | 11.05 |
Ours | 61.67 | 54.57 | 61.32 | 55.04 | 76.48 | 78.89 | 29.74 | 40.60 | 83.24 | 66.43 | 79.76 | 52.30 | 9.98 | 18.67 | 14.39 | 11.37 | |
Ours iters=1K,use_quant_input=False | 61.47 | 55.41 | 61.77 | 54.92 | 76.40 | 78.29 | 31.09 | 40.0 | 83.24 | 63.54 | 79.29 | 52.22 | 9.97 | 18.63 | 14.37 | 11.35 |
We provide a comprehensive analysis with other methods in our accuracy data section. Notably, our approach has outperformed GPTQ with a score of 30/32 and AWQ with a score of 27/32 across llamv1/llamav2/mistral-7b on W4G-1, W4G128, W3G128, W2G128. And the tuning costs are comparable.
Tips
1 Consider increasing tuning steps to achieve better results, albeit with increased tuning time. Additionally, setting 'use_quant_input' to False or adjusting 'minmax_lr' to 2.0/iters has been observed to occasionally yield improved results.
2 Leverage AutoGPTQ to run the model on GPU
from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_round import AutoRound
model_name = "facebook/opt-125m"
model = AutoModelForCausalLM.from_pretrained(
model_name, low_cpu_mem_usage=True, torch_dtype="auto", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
autoround = AutoRound(model, tokenizer, bits=4, group_size=128, scheme="asym")
autoround.quantize()
## export to autogptq
# please install auto-gptq https://github.com/AutoGPTQ/
output_dir = "/path/to/quantized_model"
autoround.export(output_dir, target="auto_gptq", use_triton=True)
# then follow auto-gptq to load the model and inference
Examples
Quantization has been enabled for various large language models. Please refer to the example readme for details.
Reference
If you find SignRound useful for your research, please cite our paper:
@article{cheng2023optimize,
title={Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs},
author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao},
journal={arXiv preprint arXiv:2309.05516},
year={2023}
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.