🔮 GPTQ - Accurate Post-Training Compression for Generative Pretrained Transformers
This repo is a extended and polished version of the original code for the paper GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers.
🔥 SOTA on LLM PTQ
- An efficient implementation of the GPTQ algorithm
- 2/3/4/8-bit quantized matrix full-precision vector product CUDA kernel
- Bug fix for old consumer-grade GPU
📥 Installation
pip install gptq
🛟 Install PyTorch
gptq requires PyTorch and GPU, and installing PyTorch with CUDA is tricky. To install PyTorch correctly, the following steps are recommended:
- run
nvcc --versionto get the version. For example, the following result means we have cuda compiler version 116
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_Mar__8_18:18:20_PST_2022
Cuda compilation tools, release 11.6, V11.6.124
Build cuda_11.6.r11.6/compiler.31057947_0
- run
pip install light-the-torchto install ltt - run
ltt install --pytorch-computation-backend=cu116 torch torchvision torchaudioto install the torch suite. Please replace the116according to your environment!
TODO
- GPTQ with CNN
Algorithm credits go to IST Austria Distributed Algorithms and Systems Lab
Metadata
Release files for gptq 0.0.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| gptq-0.0.3.tar.gz | 21.4 kB | Without provenance | Details for gptq-0.0.3.tar.gz |
Release files / gptq-0.0.3.tar.gz
| Download URL | gptq-0.0.3.tar.gz |
|---|---|
| Size | 21.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
05121652e59fd5cc9c6cf9530bb999bb4d843fdbbe81ee532e06c6f8023b812f
|
|
BLAKE2b-256 checksum How to use checksums |
25062e8e087ec1572fca200c591442ed92c4df7ac8a854ecdf32a7b8065ce14d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.8.16
|
Security for gptq 0.0.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.