Skip to main content

flashattn-amx

FlashAttention without CUDA: a FlashAttention forward pass for Apple Silicon CPUs that beats PyTorch's flash-attention CPU kernel by 1.2–2.2× in every benchmarked configuration.

It implements the same algorithm as FlashAttention (Dao et al. 2022) — tiled attention with an online softmax, O(S) extra memory instead of the O(S²) score matrix — but routes the matrix multiplies through Apple's Accelerate SGEMM, which executes on the M-series AMX matrix coprocessor, with exp vectorized via vForce and parallelism over (batch × heads × query-blocks) via Grand Central Dispatch. The kernel is ~130 lines of C.

Requires macOS on Apple Silicon. Forward pass only (inference); under autograd the PyTorch wrapper falls back to PyTorch automatically.

Install

pip install flashattn-amx

Or from source (needs Xcode command line tools):

git clone https://github.com/vineetha00/flashattn-amx
cd flashattn-amx && pip install .

Usage

PyTorch (zero-copy; registered as the custom op flashattn::forward, so it also composes with torch.compile):

import torch
from flashattn_amx.torch_ops import flash_attention, use_flash_attention

# drop-in for F.scaled_dot_product_attention (inference):
out = flash_attention(q, k, v, is_causal=True)

# or route an entire model through it — anything ineligible
# (masks, dropout, autograd, non-fp32) falls back to PyTorch:
with torch.inference_mode(), use_flash_attention():
    logits = model(idx)

use_flash_attention() patches F.scaled_dot_product_attention, so it covers models that call SDPA from Python (most hand-written and Hugging Face models), not nn.MultiheadAttention's C++ fast path.

NumPy:

import numpy as np
from flashattn_amx import flash_attention

q = np.random.randn(1, 8, 4096, 64).astype(np.float32)
out = flash_attention(q, q, q, causal=True)   # (1, 8, 4096, 64)

Block sizes are tuned for the M3's 128 KiB L1d and tunable per machine via br=/bc=.

Benchmarks

Real CUDA FlashAttention cannot run on a Mac, so the baseline is the official FlashAttention-algorithm CPU kernel that ships inside PyTorch (SDPBackend.FLASH_ATTENTION), given its best configuration (8 threads — measured faster than its 4-thread default). torch 2.15.0, float32, Apple M3 (4P + 4E). Median wall time, B=1, H=8.

S D causal ours (ms) torch flash (ms) torch naive (ms) speedup vs flash
256 64 no 0.51 0.79 2.40 1.53×
512 64 no 1.62 2.29 6.45 1.41×
1024 64 no 5.00 8.87 22.80 1.77×
2048 64 no 20.47 26.36 86.18 1.29×
4096 64 no 80.66 111.14 446.09 1.38×
8192 64 no 336.40 435.60 4296.81 1.29×
256 64 yes 0.50 0.87 2.75 1.75×
512 64 yes 1.33 2.97 8.33 2.24×
1024 64 yes 3.64 7.90 25.58 2.17×
2048 64 yes 12.98 21.67 101.64 1.67×
4096 64 yes 52.16 73.32 475.96 1.41×
8192 64 yes 203.93 244.30 5298.54 1.20×
1024 128 no 6.77 9.48 23.79 1.40×
4096 128 no 109.91 145.64 491.71 1.33×

Peak throughput 634 GFLOP/s fp32 (D=128); 10×+ faster than naive attention at long sequences. Reproduce with python bench/bench.py.

End-to-end in a real model

bench/e2e_bench.py runs a GPT-2-small-geometry transformer (12 layers × 768d × 12 heads, 162M params, fp32, CPU) and swaps every attention call to this kernel via use_flash_attention():

Seq len torch flash ours whole-model speedup
1024 836 ms 651 ms 1.28×
2048 1753 ms 1442 ms 1.22×
4096 3775 ms 3172 ms 1.19×

Max logit difference vs PyTorch: 4e-6.

Accuracy and memory

Max |error| vs a float64 exact-softmax reference is ~1e-6, the same as PyTorch's kernel (tests/test_correctness.py). Per-task scratch is Br·Bc + Br·D + 2·Br floats (~0.3 MB) regardless of sequence length; naive attention at S=8192, H=8 materializes a 2 GB score matrix.

Why it wins

  1. AMX: Accelerate's SGEMM runs on the M-series matrix coprocessor, far above what NEON SIMD (what PyTorch's CPU flash kernel uses) can reach.
  2. Cache-tuned blocks: Br=256 (128 causal), Bc=256 keeps each score tile plus accumulator inside the M3's 128 KiB L1d.
  3. Causal block skipping: key blocks entirely above the diagonal are never touched, and the diagonal tile is masked in place.
  4. All 8 cores via GCD with fine-grained (query-block) work items, so P and E cores stay saturated without oversubscription.

Layout

  • src/flashattn_amx/_core.c — the kernel
  • src/flashattn_amx/__init__.py — NumPy API
  • src/flashattn_amx/torch_ops.py — PyTorch custom op, SDPA drop-in, whole-model patch
  • tests/ — correctness vs an exact float64 reference
  • bench/ — kernel and end-to-end benchmarks

License

MIT

Metadata

Release files for flashattn-amx 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for flashattn-amx 0.1.0
File Size Uploaded Provenance Actions
flashattn_amx-0.1.0.tar.gz 11.6 kB Without provenance Details for flashattn_amx-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for flashattn-amx 0.1.0
File Interpreter ABI Platform Actions
flashattn_amx-0.1.0-cp313-cp313-macosx_12_0_arm64.whl CPython 3.13 CPython 3.13 macOS 12.0+ ARM64 Details for flashattn_amx-0.1.0-cp313-cp313-macosx_12_0_arm64.whl

Total release size: 23.9 kB

Release files / flashattn_amx-0.1.0.tar.gz

Download URL flashattn_amx-0.1.0.tar.gz
Size 11.6 kB
Tags Source
SHA-256 checksum
How to use checksums
535dafd8aa65511b37d83b5d438fa3dc7deef4789c719e3a67bb01cff99d39b0
BLAKE2b-256 checksum
How to use checksums
de0df304693a48de90e5a603730abfe8616dd12557f75ed0372e1027d0091549
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.9

Release files / flashattn_amx-0.1.0-cp313-cp313-macosx_12_0_arm64.whl

Download URL flashattn_amx-0.1.0-cp313-cp313-macosx_12_0_arm64.whl
Size 12.4 kB
Tags CPython 3.13 macOS 12.0+ ARM64
SHA-256 checksum
How to use checksums
32bd2d11b1dec08b046774f548844986da45110663cfd9f27baa734ab58cfc8f
BLAKE2b-256 checksum
How to use checksums
2287e80e849be8e41bee76663d8f8b9ff9716ffca106847f4d5fec3fa4b9720b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.9

Security for flashattn-amx 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page