This release is a pre-release and may not be stable for production use.
compressed-tensors
The compressed-tensors library extends the safetensors format, providing a versatile and efficient way to store and manage compressed tensor data. This library supports various quantization and sparsity schemes, making it a unified format for handling different model optimizations like GPTQ, AWQ, SmoothQuant, INT8, FP8, SparseGPT, and more.
Why compressed-tensors?
As model compression becomes increasingly important for efficient deployment of LLMs, the landscape of quantization and compression techniques has become increasingly fragmented.
Each method often comes with its own storage format and loading procedures, making it challenging to work with multiple techniques or switch between them.
compressed-tensors addresses this by providing a single, extensible format that can represent a wide variety of compression schemes.
- Unified Checkpoint Format: Supports various compression schemes in a single, consistent format.
- Wide Compatibility: Works with popular quantization methods like GPTQ, SmoothQuant, and FP8. See llm-compressor
- Flexible Quantization Support:
- Weight-only quantization (e.g., W4A16, W8A16, WnA16)
- Activation quantization (e.g., W8A8)
- KV cache quantization
- Non-uniform schemes (different layers can be quantized in different ways!)
- Sparsity Support: Handles both unstructured and semi-structured (e.g., 2:4) sparsity patterns.
- Open-Source Integration: Designed to work seamlessly with Hugging Face models and PyTorch.
This allows developers and researchers to easily experiment with composing different quantization methods, simplify model deployment pipelines, and reduce the overhead of supporting multiple compression formats in inference engines.
Installation
From PyPI
Stable release:
pip install compressed-tensors
Nightly release:
pip install --pre compressed-tensors
From Source
git clone https://github.com/vllm-project/compressed-tensors
cd compressed-tensors
pip install -e .
Getting started
Saving a Compressed Model with PTQ
We can use compressed-tensors to run basic post training quantization (PTQ) and save the quantized model compressed on disk
model_name = "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T"
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="cuda:0", torch_dtype="auto")
config = QuantizationConfig.parse_file("./examples/bit_packing/int4_config.json")
config.quantization_status = QuantizationStatus.CALIBRATION
apply_quantization_config(model, config)
dataset = load_dataset("ptb_text_only")["train"]
tokenizer = AutoTokenizer.from_pretrained(model_name)
def tokenize_function(examples):
return tokenizer(examples["sentence"], padding=False, truncation=True, max_length=1024)
tokenized_dataset = dataset.map(tokenize_function, batched=True)
data_loader = DataLoader(tokenized_dataset, batch_size=1, collate_fn=DefaultDataCollator())
with torch.no_grad():
for idx, sample in tqdm(enumerate(data_loader), desc="Running calibration"):
sample = {key: value.to(device) for key,value in sample.items()}
_ = model(**sample)
if idx >= 512:
break
model.apply(freeze_module_quantization)
model.apply(compress_quantized_weights)
output_dir = "./ex_llama1.1b_w4a16_packed_quantize"
compressor = ModelCompressor.from_pretrained_model(model)
compressor.compress_model(model)
model.save_pretrained(output_dir)
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file compressed_tensors-0.17.2a20260729.tar.gz.
File metadata
- Download URL: compressed_tensors-0.17.2a20260729.tar.gz
- Upload date:
- Size: 279.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a9f1455f7ec8a3c48c8d872eb03ea651996238a33f16bf59d47974a14a3a7408
|
|
| MD5 |
ba16e467969cad8b9cee3e629d103801
|
|
| BLAKE2b-256 |
5d1ce4b216cf3a7673b868f0f9d2304c3e0ea0248acd13120a1d1516a271d2a2
|
Provenance
The following attestation bundles were made for compressed_tensors-0.17.2a20260729.tar.gz:
Publisher:
upload.yml on neuralmagic/llm-compressor-testing
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
compressed_tensors-0.17.2a20260729.tar.gz -
Subject digest:
a9f1455f7ec8a3c48c8d872eb03ea651996238a33f16bf59d47974a14a3a7408 - Sigstore transparency entry: 2285993162
- Sigstore integration time:
-
Permalink:
neuralmagic/llm-compressor-testing@131f38789819782917ac764d5809bf94070f84ec -
Branch / Tag:
refs/heads/main - Owner: https://github.com/neuralmagic
-
Access:
private
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
upload.yml@131f38789819782917ac764d5809bf94070f84ec -
Trigger Event:
workflow_dispatch
-
Statement type:
File details
Details for the file compressed_tensors-0.17.2a20260729-py3-none-any.whl.
File metadata
- Download URL: compressed_tensors-0.17.2a20260729-py3-none-any.whl
- Upload date:
- Size: 219.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
f25221b58fa7796838e7ff9fd3583d95e0f16cdc1067d2ac585a10c1d17ca0ab
|
|
| MD5 |
0e9e5373450ace02eae54c49d1bd0fbd
|
|
| BLAKE2b-256 |
1052b4e9655f7106ed3060e3a9cedac251299bb98ce2e252cc01b1a3f6f3a75c
|
Provenance
The following attestation bundles were made for compressed_tensors-0.17.2a20260729-py3-none-any.whl:
Publisher:
upload.yml on neuralmagic/llm-compressor-testing
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
compressed_tensors-0.17.2a20260729-py3-none-any.whl -
Subject digest:
f25221b58fa7796838e7ff9fd3583d95e0f16cdc1067d2ac585a10c1d17ca0ab - Sigstore transparency entry: 2285993637
- Sigstore integration time:
-
Permalink:
neuralmagic/llm-compressor-testing@131f38789819782917ac764d5809bf94070f84ec -
Branch / Tag:
refs/heads/main - Owner: https://github.com/neuralmagic
-
Access:
private
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
upload.yml@131f38789819782917ac764d5809bf94070f84ec -
Trigger Event:
workflow_dispatch
-
Statement type:
