Fast Hadamard Transform in CUDA, with a PyTorch interface

Features:

  • Support fp32, fp16, bf16, for dimension up to 32768.
  • Implicitly pad with zeros if dimension is not a power of 2.

Installation

git clone https://github.com/Dao-AILab/fast-hadamard-transform.git fast-hadamard-transform
cd fast-hadamard-transform
pip install -v .

How to use

from fast_hadamard_transform import hadamard_transform

def hadamard_transform(x, scale=1.0):
    """
    Arguments:
        x: (..., dim)
        scale: float. Multiply the output by this number.
    Returns:
        out: (..., dim)

    Multiply each row of x by the Hadamard transform matrix.
    Equivalent to F.linear(x, torch.tensor(scipy.linalg.hadamard(dim))) * scale.
    If dim is not a power of 2, we implicitly pad x with zero so that dim is the next power of 2.
    """

Speed

Benchmarked on A100, for not too small batch size, compared to memcpy (torch.clone), which is a lower bound for the time taken as we'd need to read inputs from GPU memory and write output to GPU memory anyway.

Data type Dimension Time taken vs memcpy
fp16/bf16 <= 512 1.0x
512 - 8192 <= 1.2x
16384 1.3x
32768 1.8x
fp32 <= 8192 1.0x
16384 1.1x
32768 1.2x

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for lightseek from gravatar.com lightseek

Credits

Author: Tri Dao

License

BSD License

Requires

Python >=3.7

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp313-cp313-manylinux_2_28_x86_64.whl (19.3 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.28+ x86-64

tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp313-cp313-manylinux_2_28_aarch64.whl (19.3 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.28+ ARM64

tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp312-cp312-manylinux_2_28_x86_64.whl (19.3 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.28+ x86-64

tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp312-cp312-manylinux_2_28_aarch64.whl (19.3 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.28+ ARM64

tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp311-cp311-manylinux_2_28_x86_64.whl (19.3 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.28+ x86-64

tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp311-cp311-manylinux_2_28_aarch64.whl (19.3 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.28+ ARM64

tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp310-cp310-manylinux_2_28_x86_64.whl (19.3 MB view details)

Uploaded CPython 3.10manylinux: glibc 2.28+ x86-64

tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp310-cp310-manylinux_2_28_aarch64.whl (19.3 MB view details)

Uploaded CPython 3.10manylinux: glibc 2.28+ ARM64

File details

Details for the file tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp313-cp313-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp313-cp313-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 3d3668018f6fbf4053dae4c02856bf20eb43ccfd371cb1c27be1dd1182a2e21e
MD5 6a5643b6a0f490419cd1f898a4f61dce
BLAKE2b-256 4f4e69af1931aac46e8fa46a745b101d14c8fb0de7d6da505f216a2880e68cc0

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp313-cp313-manylinux_2_28_x86_64.whl:

Publisher: tokenspeed-fast-hadamard-transform.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp313-cp313-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp313-cp313-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 9943220b401077c186acfee75119092e7a16a95670f5f710b4b8f99af13cbcca
MD5 87b21eef790318a5d0ce98007cbdc90a
BLAKE2b-256 e66b58ada2080fcbc42195c37eb17b8ec8720d301c005c4d97ed5388aaa35ebc

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp313-cp313-manylinux_2_28_aarch64.whl:

Publisher: tokenspeed-fast-hadamard-transform.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp312-cp312-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp312-cp312-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 0f5b1fd0461058fa466ebdfea1eb99aeb7c6e981fe992c66485b1ea331a71995
MD5 fbaf4d23d874c52c9bd35a9e034d6caa
BLAKE2b-256 9693ad7b61f437277f1bc0eb4902fa77fc3cb8af1e1fcd0e02b71ebeed4eb6f0

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp312-cp312-manylinux_2_28_x86_64.whl:

Publisher: tokenspeed-fast-hadamard-transform.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp312-cp312-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp312-cp312-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 22d49783ce9316e25aa415ad71df4f3aa962b636e1e9a45b8a46746e57cb3073
MD5 f8f80db78fd6514371f699e41f852b96
BLAKE2b-256 750306a9374ae220cbae34939f3e78ef31bd9c3064ae14f85e8ee08ab28a19b6

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp312-cp312-manylinux_2_28_aarch64.whl:

Publisher: tokenspeed-fast-hadamard-transform.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp311-cp311-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp311-cp311-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 b7034c074778d7dd1e2af5a0b5c4039144b5c1565433befdcf308e25ca595aef
MD5 f3934a771e0e90e9973dbbdf31044e92
BLAKE2b-256 ac87967a65091c63eab5e1c06e717680d42d3feca7041d85a44e91a3fc1f0187

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp311-cp311-manylinux_2_28_x86_64.whl:

Publisher: tokenspeed-fast-hadamard-transform.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp311-cp311-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp311-cp311-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 ea28f5724e62094f1a3592064447113c7ddb01b9832fcc75af7de10852c3af2e
MD5 7cf342575045087775318af9a3be99d7
BLAKE2b-256 0ccb1ea4e963ea17b2f42bfc9d40fa44ab489b64d8d70cb7a9508e3c5c4846ff

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp311-cp311-manylinux_2_28_aarch64.whl:

Publisher: tokenspeed-fast-hadamard-transform.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp310-cp310-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp310-cp310-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 35c0c7c55f13d70272bf5fa22b2be741bfe6fca40742218e1f4009acd9199f8b
MD5 d774e15cdac7e5d90dc8a06711446c30
BLAKE2b-256 69e218baa829133fd621db7f05f37bb1bacc66161ab28ae0a6989813e20c55a6

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp310-cp310-manylinux_2_28_x86_64.whl:

Publisher: tokenspeed-fast-hadamard-transform.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp310-cp310-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp310-cp310-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 a55b51a174408b57ab6535f52140348ee76f1c624e2b9ce191070b057b39d2f1
MD5 e857d8d290bf2a7ba28848680124abaa
BLAKE2b-256 41f45503e67f78024b5c70d039d48d0260d533fd71654312317c46d54891bcd6

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_fast_hadamard_transform-1.1.0.post20260810-cp310-cp310-manylinux_2_28_aarch64.whl:

Publisher: tokenspeed-fast-hadamard-transform.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

1.1.0.post20260810 This release

8 files

1.1.0.post20260310

8 files

1.1.0.post20251110

8 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page