Fast Hadamard Transform in CUDA, with a PyTorch interface

Features:

  • Support fp32, fp16, bf16, for dimension up to 32768.
  • Implicitly pad with zeros if dimension is not a power of 2.

Installation

git clone https://github.com/Dao-AILab/fast-hadamard-transform.git fast-hadamard-transform
cd fast-hadamard-transform
pip install -v .

How to use

from fast_hadamard_transform import hadamard_transform

def hadamard_transform(x, scale=1.0):
    """
    Arguments:
        x: (..., dim)
        scale: float. Multiply the output by this number.
    Returns:
        out: (..., dim)

    Multiply each row of x by the Hadamard transform matrix.
    Equivalent to F.linear(x, torch.tensor(scipy.linalg.hadamard(dim))) * scale.
    If dim is not a power of 2, we implicitly pad x with zero so that dim is the next power of 2.
    """

Speed

Benchmarked on A100, for not too small batch size, compared to memcpy (torch.clone), which is a lower bound for the time taken as we'd need to read inputs from GPU memory and write output to GPU memory anyway.

Data type Dimension Time taken vs memcpy
fp16/bf16 <= 512 1.0x
512 - 8192 <= 1.2x
16384 1.3x
32768 1.8x
fp32 <= 8192 1.0x
16384 1.1x
32768 1.2x

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.1.0.post20260810)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for lightseek from gravatar.com lightseek

Credits

Author: Tri Dao

License

BSD License

Requires

Python >=3.7

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp313-cp313-manylinux_2_28_x86_64.whl (19.1 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.28+ x86-64

tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp313-cp313-manylinux_2_28_aarch64.whl (19.1 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.28+ ARM64

tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp312-cp312-manylinux_2_28_x86_64.whl (19.1 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.28+ x86-64

tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp312-cp312-manylinux_2_28_aarch64.whl (19.1 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.28+ ARM64

tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp311-cp311-manylinux_2_28_x86_64.whl (19.1 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.28+ x86-64

tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp311-cp311-manylinux_2_28_aarch64.whl (19.1 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.28+ ARM64

tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp310-cp310-manylinux_2_28_x86_64.whl (19.1 MB view details)

Uploaded CPython 3.10manylinux: glibc 2.28+ x86-64

tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp310-cp310-manylinux_2_28_aarch64.whl (19.1 MB view details)

Uploaded CPython 3.10manylinux: glibc 2.28+ ARM64

File details

Details for the file tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp313-cp313-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp313-cp313-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 7ad767f97ab649ae108e0e5b22d9d428ea84f0a232efeb5cf1fbfdafa096e7d0
MD5 490582bc0837861988430ec294393b65
BLAKE2b-256 391ed41d2af24a146d72f493cfe9688caaf523e6f60f1803f9c01b1cce6128fb

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp313-cp313-manylinux_2_28_x86_64.whl:

Publisher: tokenspeed-fast-hadamard-transform.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp313-cp313-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp313-cp313-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 e20daf925e179c8387639bc14e2ef509c6eda0d34073e545d814c3b650e35955
MD5 f09d53b37e0550c8e43e1ee596df918a
BLAKE2b-256 0ee703f0ab2a4d7ba323de5f91fe7f4e6d0f25ad2fcaeae40f7ca3298c5f414b

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp313-cp313-manylinux_2_28_aarch64.whl:

Publisher: tokenspeed-fast-hadamard-transform.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp312-cp312-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp312-cp312-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 4ad277a0df2e3f67bd141f62de347b810852e5acb0dc35b5c21c9ccc119b3867
MD5 da0eca1a65bb9cd588fb5e9acaa534b8
BLAKE2b-256 b26be30345f7a9c7f16d644f78910dc217a76db06765de9618ecb372f53395ae

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp312-cp312-manylinux_2_28_x86_64.whl:

Publisher: tokenspeed-fast-hadamard-transform.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp312-cp312-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp312-cp312-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 75c2de7978f6ad2174ad7223300291fe5692f8c527a3f748f677cfca6b53d85a
MD5 03d943338f437a2a022d477c56176e47
BLAKE2b-256 3ed9de7d14b97dd1ecf25ee494dd9a98c5ff48ee2d064eb83b0d6d152ffd3650

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp312-cp312-manylinux_2_28_aarch64.whl:

Publisher: tokenspeed-fast-hadamard-transform.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp311-cp311-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp311-cp311-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 c32ed7ffd989fa57aebe1e701ea0b0fa35021cae57db139829c838ac16542fc8
MD5 bef9c4fa0b9035920f3f85dd01f35074
BLAKE2b-256 e464a4467d751968d5c1c0b7047ea5fe9b5fdc1c89b11bb0cf3e9cb37f7a3132

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp311-cp311-manylinux_2_28_x86_64.whl:

Publisher: tokenspeed-fast-hadamard-transform.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp311-cp311-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp311-cp311-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 28e525d819969a85f08f0d735945b4ae49e1c680a55fa3cf78dcac63b9247521
MD5 7ac159595f7f1c9636abce78faed3b02
BLAKE2b-256 9c36db80cb16b38cebec2ca9cb8b43038c3faf8f2783c15702e7e953ebe72182

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp311-cp311-manylinux_2_28_aarch64.whl:

Publisher: tokenspeed-fast-hadamard-transform.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp310-cp310-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp310-cp310-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 5bab932d5c50f5a15ad899bd136a4b448beff3a3e3a2fb900f24682e45ce1529
MD5 12f83e8415bc743673d4682327b02d24
BLAKE2b-256 b0fca38ee858e3a0f9974ef5e175f5dea915d7a96e1f74489310ca24778aa22e

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp310-cp310-manylinux_2_28_x86_64.whl:

Publisher: tokenspeed-fast-hadamard-transform.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp310-cp310-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp310-cp310-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 d78f01263a64b9c06776cc485e9c38f5cec9abdb530a9dbb21fa05262a05cd38
MD5 175bcb1414ba09a08df02f329b2a684f
BLAKE2b-256 0c55cd154d02329b52add98eec0157b6d1b9b5724fea6adf1790c391feac4c87

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_fast_hadamard_transform-1.1.0.post20260310-cp310-cp310-manylinux_2_28_aarch64.whl:

Publisher: tokenspeed-fast-hadamard-transform.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

1.1.0.post20260810

8 files

This release

1.1.0.post20260310 This release

8 files

1.1.0.post20251110

8 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page