🔮 GPTQ - Accurate Post-Training Compression for Generative Pretrained Transformers

This repo is a extended and polished version of the original code for the paper GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers.

🔥 SOTA on LLM PTQ

An efficient implementation of the GPTQ algorithm
2/3/4/8-bit quantized matrix full-precision vector product CUDA kernel
Bug fix for old consumer-grade GPU

📥 Installation

pip install gptq

🛟 Install PyTorch

gptq requires PyTorch and GPU, and installing PyTorch with CUDA is tricky. To install PyTorch correctly, the following steps are recommended:

run nvcc --version to get the version. For example, the following result means we have cuda compiler version 116

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_Mar__8_18:18:20_PST_2022
Cuda compilation tools, release 11.6, V11.6.124
Build cuda_11.6.r11.6/compiler.31057947_0

run pip install light-the-torch to install ltt
run ltt install --pytorch-computation-backend=cu116 torch torchvision torchaudio to install the torch suite. Please replace the 116 according to your environment!

TODO

GPTQ with CNN

Algorithm credits go to IST Austria Distributed Algorithms and Systems Lab

Metadata

Release files for gptq 0.0.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for gptq 0.0.3
File	Size	Uploaded
gptq-0.0.3.tar.gz	21.4 kB	Mar 23, 2023	Details

Release files / gptq-0.0.3.tar.gz

Download URL	gptq-0.0.3.tar.gz
Size	21.4 kB
Tags	Source
SHA-256 checksum How to use checksums	`05121652e59fd5cc9c6cf9530bb999bb4d843fdbbe81ee532e06c6f8023b812f`
BLAKE2b-256 checksum How to use checksums	`25062e8e087ec1572fca200c591442ed92c4df7ac8a854ecdf32a7b8065ce14d`
Upload date	Mar 23, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.8.16`