hashcodecs

hashcodecs provides runtime-dispatched SIMD Base64 codecs and fast, reference-compatible MurmurHash3 functions for Rust and Python.

Design

• Base64 is implemented in this crate with runtime AVX-512 VBMI, AVX2, SSE4.1, SSSE3, and AArch64 NEON dispatch. Portable wheels select the best available SIMD backend and fall back to scalar code otherwise.
• Rust callers can reuse their own output buffers through the *_into APIs. Every backend writes exactly the documented prefix; it never depends on spare capacity after the destination slice.
• Rust-owned results use the mimalloc global allocator.
• Every MurmurHash3 variant uses runtime-dispatched AVX2 or SSE4.1 pre-mixing where its measured crossover beats the scalar loop. The x64 128-bit AVX2 kernel also selects BMI2 rotations when available. Ordered state transitions remain reference-compatible, and every variant has a portable scalar fallback.
• The Python Base64 surface follows the familiar base64 names. Use import hashcodecs.base64 as base64 when replacing standard Base64 calls.
• Large Python calls release the GIL. Strict decoding and valid default-mode decoding write directly into the returned bytes; malformed default-mode input uses a CPython-compatible lenient fallback.

Install

pip3 install hashcodecs

Usage

Rust

let encoded = hashcodecs::b64encode(b"hello");
assert_eq!(encoded, "aGVsbG8=");

let mut output = [0_u8; 8];
let written = hashcodecs::b64encode_into(b"hello", &mut output).unwrap();
assert_eq!(&output[..written], b"aGVsbG8=");
assert_eq!(hashcodecs::murmur3_x86_32(b"hello", 0), 0x248b_fa47);

Python

import hashcodecs.base64 as base64
from hashcodecs import murmur3_32, murmur3_x64_128

assert base64.b64encode(b"hello") == b"aGVsbG8="
assert base64.b64decode(b"aGVsbG8=") == b"hello"
assert murmur3_32(b"hello") == 0x248BFA47

hasher = murmur3_x64_128(seed=42)
hasher.update(b"hello")
assert hasher.hexdigest() == hasher.digest().hex()

Build an installable wheel and source distribution with:

uv build

Benchmark

Environment: Windows 10 x64 and Intel Core Ultra 7 265K.

Conditions: clean builds, one pinned logical CPU, single-threaded execution, and output allocation included. Higher is better.

Run Locally

Comparison crates are development-only dependencies and are not included in consumer builds.

cargo bench --bench base64
cargo bench --bench murmur3
uv sync --group benchmark --no-install-project
uv run --no-project python benchmarks/python_base64.py
uv run --no-project python benchmarks/python_murmur3.py
uv run --no-project python benchmarks/python_murmur3.py --incremental

To time only this implementation:

cargo bench --bench base64 -- hashcodecs
cargo bench --bench murmur3 -- hashcodecs
uv run --no-project python benchmarks/python_base64.py --hashcodecs-only
uv run --no-project python benchmarks/python_murmur3.py --hashcodecs-only
uv run --no-project python benchmarks/python_murmur3.py --incremental --hashcodecs-only

Base64: Rust

MurmurHash3: Rust

Base64: Python

Python decoding uses validate=True, and hashcodecs passes bytes directly into Rust without an input copy.

MurmurHash3: Python

SIMD References

The AVX2 block structure uses the 24-byte encode and 32-byte decode arrangement described in Faster Base64 Encoding and Decoding using AVX2 Instructions. The AVX-512 VBMI path processes 48-byte encode and 64-byte decode blocks with cross-lane byte permutations, while AArch64 uses 48-byte NEON encoding and 64-byte NEON decoding blocks. The Rust 1.89 MSRV is the first stable release that exposes the required AVX-512 intrinsics.