hashcodecs

hashcodecs provides runtime-dispatched SIMD Base64 codecs and fast, reference-compatible MurmurHash3 functions for Rust and Python.

Design

• Base64 is implemented in this crate with runtime AVX-512 VBMI, AVX2, SSE4.1, SSSE3, and AArch64 NEON dispatch. Portable wheels select the best available SIMD backend and fall back to scalar code otherwise.
• Rust callers can reuse their own output buffers through the *_into APIs. Every backend writes exactly the documented prefix; it never depends on spare capacity after the destination slice.
• Rust-owned results use the mimalloc global allocator.
• Every MurmurHash3 variant uses runtime-dispatched AVX2 or SSE4.1 pre-mixing where its measured crossover beats the scalar loop. The x64 128-bit AVX2 kernel also selects BMI2 rotations when available. Ordered state transitions remain reference-compatible, and every variant has a portable scalar fallback.
• The Python Base64 surface follows the familiar base64 names. Use import hashcodecs.base64 as base64 when replacing standard Base64 calls.
• Python callers can reuse a bytearray through the *_into APIs when returned bytes ownership is unnecessary.
• Large Python calls release the GIL. Strict decoding and valid default-mode decoding write directly into the returned bytes; malformed default-mode input uses a CPython-compatible lenient fallback.

Install

pip3 install hashcodecs

Usage

Rust

let encoded = hashcodecs::b64encode(b"hello");
assert_eq!(encoded, "aGVsbG8=");

let mut output = [0_u8; 8];
let written = hashcodecs::b64encode_into(b"hello", &mut output).unwrap();
assert_eq!(&output[..written], b"aGVsbG8=");
assert_eq!(hashcodecs::murmur3_x86_32(b"hello", 0), 0x248b_fa47);

Python

import hashcodecs.base64 as base64
from hashcodecs import murmur3_32, murmur3_x64_128

assert base64.b64encode(b'hello') == b'aGVsbG8='
assert base64.b64decode(b'aGVsbG8=') == b'hello'
assert murmur3_32(b'hello') == 0x248BFA47

output = bytearray(8)
written = base64.b64encode_into(b'hello', output)
assert output[:written] == b'aGVsbG8='

hasher = murmur3_x64_128(seed=42)
hasher.update(b'hello')
assert hasher.hexdigest() == hasher.digest().hex()

Build an installable wheel and source distribution with:

uv build

Benchmark

Environment: Windows 10 x64 and Intel Core Ultra 7 265K.

Conditions: clean builds, one pinned logical CPU, single-threaded execution, 50 Rust samples, and nine Python samples. Returned-output allocation is included except in the reusable-buffer table. Higher is better.

Run Locally

Comparison crates are development-only dependencies and are not included in consumer builds.

cargo bench --bench base64
cargo bench --bench murmur3
uv sync --group benchmark --no-install-project
uv run --no-project python benchmarks/python_base64.py
uv run --no-project python benchmarks/python_base64.py --into
uv run --no-project python benchmarks/python_murmur3.py
uv run --no-project python benchmarks/python_murmur3.py --incremental

To time only this implementation:

cargo bench --bench base64 -- hashcodecs
cargo bench --bench murmur3 -- hashcodecs
uv run --no-project python benchmarks/python_base64.py --hashcodecs-only
uv run --no-project python benchmarks/python_murmur3.py --hashcodecs-only
uv run --no-project python benchmarks/python_murmur3.py --incremental --hashcodecs-only

Base64: Rust

MurmurHash3: Rust

Base64: Python

Python decoding uses validate=True, and hashcodecs passes bytes directly into Rust without an input copy.

Reusable Python Buffers

MurmurHash3: Python

SIMD References

The SIMD implementation follows the approach described in Faster Base64 Encoding and Decoding using AVX2 Instructions, with AVX-512 VBMI and AArch64 NEON backends selected automatically when available.