Fast Hadamard Transform in CUDA, with a PyTorch interface
Features:
- Support fp32, fp16, bf16, for dimension up to 32768.
- Implicitly pad with zeros if dimension is not a power of 2.
Installation
git clone https://github.com/Dao-AILab/fast-hadamard-transform.git fast-hadamard-transform
cd fast-hadamard-transform
pip install -v .
How to use
from fast_hadamard_transform import hadamard_transform
def hadamard_transform(x, scale=1.0):
"""
Arguments:
x: (..., dim)
scale: float. Multiply the output by this number.
Returns:
out: (..., dim)
Multiply each row of x by the Hadamard transform matrix.
Equivalent to F.linear(x, torch.tensor(scipy.linalg.hadamard(dim))) * scale.
If dim is not a power of 2, we implicitly pad x with zero so that dim is the next power of 2.
"""
Speed
Benchmarked on A100, for not too small batch size, compared to memcpy (torch.clone), which is a lower bound for the time taken as we'd need to read inputs from GPU memory and write output to GPU memory anyway.
| Data type | Dimension | Time taken vs memcpy |
|---|---|---|
| fp16/bf16 | <= 512 | 1.0x |
| 512 - 8192 | <= 1.2x | |
| 16384 | 1.3x | |
| 32768 | 1.8x | |
| fp32 | <= 8192 | 1.0x |
| 16384 | 1.1x | |
| 32768 | 1.2x |
1 maintainer
Metadata
1 maintainer
Release files for tokenspeed-fast-hadamard-transform 1.1.0.post20260920
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distributions (wheels)
Total release size: 154.3 MB
Release files / tokenspeed_fast_hadamard_transform-1.1.0.post20260920-cp313-cp313-manylinux_2_28_x86_64.whl
| Download URL | tokenspeed_fast_hadamard_transform-1.1.0.post20260920-cp313-cp313-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 19.3 MB |
| Tags | CPython 3.13 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
48bf4dfc3e29bab8987950420cc2f2d89660600e00cdfb1ea0ed32e7f6fb6c46
|
|
BLAKE2b-256 checksum How to use checksums |
6901499273b40d569a672897b46c90d5581259b55d5ec8fce06497797ae3620f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / tokenspeed_fast_hadamard_transform-1.1.0.post20260920-cp313-cp313-manylinux_2_28_aarch64.whl
| Download URL | tokenspeed_fast_hadamard_transform-1.1.0.post20260920-cp313-cp313-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 19.3 MB |
| Tags | CPython 3.13 Linux glibc 2.28+ ARM64 |
|
SHA-256 checksum How to use checksums |
46296fb801aa2d44e8deeeb1f0138540b5784ab9f33e02a68d29dffc80de7c02
|
|
BLAKE2b-256 checksum How to use checksums |
db1b0d4338b60267fdd3aa00c5db35bc7e9de89a2a480f4f38892ad21d9fb45c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / tokenspeed_fast_hadamard_transform-1.1.0.post20260920-cp312-cp312-manylinux_2_28_x86_64.whl
| Download URL | tokenspeed_fast_hadamard_transform-1.1.0.post20260920-cp312-cp312-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 19.3 MB |
| Tags | CPython 3.12 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
2af87a2e3e3a73d5b61f28b3edcb6d4e6733a56642242b59e103a59f91506fd6
|
|
BLAKE2b-256 checksum How to use checksums |
c18b2291b115d84663aa61e6482041783bab99a0330e8bb8199743c406f95d9a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / tokenspeed_fast_hadamard_transform-1.1.0.post20260920-cp312-cp312-manylinux_2_28_aarch64.whl
| Download URL | tokenspeed_fast_hadamard_transform-1.1.0.post20260920-cp312-cp312-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 19.3 MB |
| Tags | CPython 3.12 Linux glibc 2.28+ ARM64 |
|
SHA-256 checksum How to use checksums |
87910dbbabb5f3d9546fca5f44ceb2cffa92ce95f4b8862884fa3e9aa49fedc1
|
|
BLAKE2b-256 checksum How to use checksums |
9aaf4e5ee871030b345e7cb7ce596894ad6a712b427926ba0992594ca38df291
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / tokenspeed_fast_hadamard_transform-1.1.0.post20260920-cp311-cp311-manylinux_2_28_x86_64.whl
| Download URL | tokenspeed_fast_hadamard_transform-1.1.0.post20260920-cp311-cp311-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 19.3 MB |
| Tags | CPython 3.11 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
54b4991b6a0d0c422bd3244e5f23defdba429d2516e121a19ed625a91376fbad
|
|
BLAKE2b-256 checksum How to use checksums |
09648e43b39743af841ff50eb429d5c1e2c0268e92c6a67e979da6899a3a763c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / tokenspeed_fast_hadamard_transform-1.1.0.post20260920-cp311-cp311-manylinux_2_28_aarch64.whl
| Download URL | tokenspeed_fast_hadamard_transform-1.1.0.post20260920-cp311-cp311-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 19.3 MB |
| Tags | CPython 3.11 Linux glibc 2.28+ ARM64 |
|
SHA-256 checksum How to use checksums |
d5999565e75cdb922ff45c9105d37be7da882ca0bfe09dc5bdb7a1cdc8ba13cb
|
|
BLAKE2b-256 checksum How to use checksums |
bda6c5aa819afc9c8dbe15f53d795dc899f8f2a6ce8d5a5e069d30de58d3be5d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / tokenspeed_fast_hadamard_transform-1.1.0.post20260920-cp310-cp310-manylinux_2_28_x86_64.whl
| Download URL | tokenspeed_fast_hadamard_transform-1.1.0.post20260920-cp310-cp310-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 19.3 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
06b07aa394310909e1a98878a1c9c20d7746246a29326ef5c350f34b3816af6c
|
|
BLAKE2b-256 checksum How to use checksums |
952037fc7da37c4d7db8ddb81f60dedf7b1c51a0b134cab8bc3cfab621855bf2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / tokenspeed_fast_hadamard_transform-1.1.0.post20260920-cp310-cp310-manylinux_2_28_aarch64.whl
| Download URL | tokenspeed_fast_hadamard_transform-1.1.0.post20260920-cp310-cp310-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 19.3 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ ARM64 |
|
SHA-256 checksum How to use checksums |
8ede61ca190c0f41b05c936882923a86bba611a25049df3130d4a592765707a4
|
|
BLAKE2b-256 checksum How to use checksums |
fb70e9aff6739f47f0221e6be7a4b019e64fa2a5ec89cf22f1b1187af383fcd3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
