compressed-tensors
The compressed-tensors library extends the safetensors format, providing a versatile and efficient way to store and manage compressed tensor data. This library supports various compression schemes, making it a unified format for handling models compressed with algorithms like GPTQ, AWQ, SmoothQuant, and SparseGPT, across formats like INT8, FP8, NVFP4, MXFP4, MXFP8, and more.
Why compressed-tensors?
As model compression becomes increasingly important for efficient deployment of LLMs, the landscape of quantization and compression techniques has become increasingly fragmented.
Each method often comes with its own storage format and loading procedures, making it challenging to work with multiple techniques or switch between them.
compressed-tensors addresses this by providing a single, extensible format that can represent a wide variety of compression schemes.
- Unified Checkpoint Format: Supports various compression schemes in a single, consistent format.
- Wide Compatibility: Works with popular quantization methods like GPTQ, SmoothQuant, AWQ, AutoRound, etc. See llm-compressor
- Flexible Quantization Support:
- Activation quantization
- Mixed precision
- Low/arbitrary-bit
- KV cache quantization
- Non-uniform schemes (different layers can be quantized in different ways!)
- Sparsity Support: Handles both unstructured and semi-structured (e.g., 2:4) sparsity patterns.
- Transform Support: Rotation-based quantization techniques (Hadamard, random Hadamard, random matrix transforms).
- Checkpoint Conversion: Convert between formats like AutoAWQ, ModelOpt NVFP4, FP8 block, and compressed-tensors.
- Model Offloading: Transparent CPU/disk/distributed offloading for models larger than available VRAM.
- Open-Source Integration: Designed to work seamlessly with Hugging Face models, PyTorch, vLLM, and SGLang.
This allows developers and researchers to easily experiment with composing different quantization methods, simplify model deployment pipelines, and reduce the overhead of supporting multiple compression formats in inference engines.
Installation
From PyPI
Stable release:
pip install compressed-tensors
Nightly release:
pip install --pre compressed-tensors
From Source
git clone https://github.com/vllm-project/compressed-tensors
cd compressed-tensors
pip install -e .
Development
Install the development dependencies and run the linting, formatting, and type checks:
pip install -e .[dev]
make quality # check
make style # auto-fix
Pre-commit Hooks
We provide pre-commit hooks that run the same checks as make quality (plus a DCO sign-off hook) before each commit, so problems are caught locally instead of in CI. After installing the [dev] dependencies, enable them once per clone:
pre-commit install
The hooks then run automatically on git commit. To run them against all files on demand:
pre-commit run --all-files
To bypass the hooks for a single commit, use git commit --no-verify; to skip one hook, prefix the command with SKIP=<hook-id> (e.g. SKIP=flake8).
Getting Started
Compressing a Model to MXFP4
The following example loads Llama 3 8B, applies round-to-nearest (RTN) MXFP4 weight quantization, compresses the weights, and saves the result. No calibration data is needed — scales are computed directly from the weights.
model_name = "meta-llama/Meta-Llama-3-8B"
device = "cuda:0" if torch.cuda.is_available() else "cpu"
# Load the model
model = AutoModelForCausalLM.from_pretrained(
model_name, device_map=device, torch_dtype="auto"
)
# Set-up the quantization config. This defines:
# 1. What quantization scheme we're applying and to which layers
# 2. Any layers that should be ignored
# In this case, all the Linear layers are targeted, apart from the lm_head
config = QuantizationConfig(
config_groups={"MXFP4": ["Linear"]},
ignore=["lm_head"],
)
# Apply the config to the model. This step uses the config to define
# the quantization parameters (such as the scales) for the targeted layers
# and attaches a QuantizationScheme which defines how the weights and activations
# should be quantized (e.g number of bits, group or block sizes, etc)
apply_quantization_config(model, config)
# Compute weight scales using round-to-nearest quantization
for name, module in model.named_modules():
# Only target layers with a QuantizationScheme attached
scheme = getattr(module, "quantization_scheme", None)
if scheme is None or scheme.weights is None:
continue
weight = module.weight.data
args = scheme.weights
# MXFP4 uses group-wise quantization for its weights, with group_size 32
group_size = args.group_size
if group_size is not None and group_size > 0:
reshaped = weight.unflatten(-1, (math.ceil(weight.shape[-1] / group_size), group_size))
min_val = reshaped.amin(dim=-1)
max_val = reshaped.amax(dim=-1)
else:
min_val, max_val = torch.aminmax(weight)
# Calculate the quantization parameters, such as the weight scale, using the min and max values
scale, _ = calculate_qparams(min_val, max_val, args)
# Update the parameters attached to the module based on the calculated value
# In this case, we update the `weight_scale` attached to the targeted linear layers
update_offload_parameter(module, "weight_scale", scale)
output_dir = "./Meta-Llama-3-8B-MXFP4"
# set-up a compressor
compressor = ModelCompressor.from_pretrained_model(model)
# Compress the model using the calibrated scales and save it using the mxfp4-pack-quantized format.
# This format defines the weight packing, which can be seamlessly loaded through vLLM.
compressor.compress_model(model)
model.save_pretrained(output_dir)
# Update the model's config with the relevant compressed-tensors details, illustrated below.
compressor.update_config(output_dir)
Once done, the config.json will have the following quantization_config:
"quantization_config": {
"config_groups": {
"group_0": {
"format": "mxfp4-pack-quantized",
"input_activations": {
"actorder": null,
"block_structure": null,
"dynamic": true,
"group_size": 32,
"num_bits": 4,
"observer": null,
"observer_kwargs": {},
"scale_dtype": "torch.uint8",
"strategy": "group",
"symmetric": true,
"type": "float",
"zp_dtype": null
},
"output_activations": null,
"targets": [
"Linear"
],
"weights": {
"actorder": null,
"block_structure": null,
"dynamic": false,
"group_size": 32,
"num_bits": 4,
"observer": "memoryless_minmax",
"observer_kwargs": {},
"scale_dtype": "torch.uint8",
"strategy": "group",
"symmetric": true,
"type": "float",
"zp_dtype": null
}
}
},
"format": "mxfp4-pack-quantized",
"global_compression_ratio": null,
"ignore": [
"lm_head"
],
"kv_cache_scheme": null,
"quant_method": "compressed-tensors",
"quantization_status": "compressed",
"sparsity_config": {},
"transform_config": {},
"version": "0.18.1.dev0+gac8e2ba.d20260813"
},
See examples/ for more examples including quantization with calibration and checkpoint conversion (examples/convert_checkpoint/).
Citation
If you find compressed-tensors useful in your research or projects, please consider citing it:
@software{compressedtensors2024,
title={{compressed-tensors}},
author={Red Hat AI and vLLM Project},
year={2024},
month={4},
url={https://github.com/vllm-project/compressed-tensors},
}
1 maintainer
Release files for compressed-tensors 0.19.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| compressed_tensors-0.19.0.tar.gz | 336.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| compressed_tensors-0.19.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 576.5 kB
Release files / compressed_tensors-0.19.0.tar.gz
| Download URL | compressed_tensors-0.19.0.tar.gz |
|---|---|
| Size | 336.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c66f72f121df8970722f708bc2ca719dc1c869004d4fbf6294403b2de1049749
|
|
BLAKE2b-256 checksum How to use checksums |
a59d7ac32c1754d64b9cedeb962b65a2e4282839736f42984df42a737bc1e7f0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 21, 2026.
Transparency logRelease files / compressed_tensors-0.19.0-py3-none-any.whl
| Download URL | compressed_tensors-0.19.0-py3-none-any.whl |
|---|---|
| Size | 240.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
5c2dcfd9cd820521906e23e096a51d68755cb88b67e19b1a5c9e11fa69637fb2
|
|
BLAKE2b-256 checksum How to use checksums |
c3a44bb391f956a414a5a0223337774ade6def27dce847bca59e4880bc146609
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 21, 2026.
Transparency log
