hashcodecs

hashcodecs provides runtime-dispatched SIMD Base64 codecs and fast, reference-compatible MurmurHash3 functions for Rust and Python.

Design

• Base64 is implemented in this crate with runtime AVX2 and SSSE3 dispatch. One portable wheel supports Intel and AMD CPUs, selecting SIMD where the OS and CPU allow it and a scalar fallback otherwise.
• Rust callers can reuse their own output buffers through the *_into APIs. Every backend writes exactly the documented prefix; it never depends on spare capacity after the destination slice.
• Rust-owned results use the mimalloc global allocator.
• MurmurHash3 uses direct little-endian block loads and tight scalar loops. Its state transitions are loop-carried, so the scalar implementation is faster than forcing wide-vector instructions for these hashes.
• The Python Base64 surface follows the familiar base64 names. Use import hashcodecs.base64 as base64 when replacing standard Base64 calls.
• Large Python calls release the GIL. Strict decoding and valid default-mode decoding write directly into the returned bytes; malformed default-mode input uses a CPython-compatible lenient fallback.

Usage

Rust

let encoded = hashcodecs::b64encode(b"hello");
assert_eq!(encoded, "aGVsbG8=");

let mut output = [0_u8; 8];
let written = hashcodecs::b64encode_into(b"hello", &mut output).unwrap();
assert_eq!(&output[..written], b"aGVsbG8=");
assert_eq!(hashcodecs::murmur3_x86_32(b"hello", 0), 0x248b_fa47);

Python

import hashcodecs.base64 as base64
from hashcodecs import murmur3_32

assert base64.b64encode(b"hello") == b"aGVsbG8="
assert base64.b64decode(b"aGVsbG8=") == b"hello"
assert murmur3_32(b"hello") == 0x248BFA47

Build an installable wheel and source distribution with:

uv build

Benchmark

Environment: Windows 10 x64 and Intel Core Ultra 7 265K.

Conditions: clean builds, one pinned logical CPU, single-threaded execution, and output allocation included. Higher is better.

Run Locally

Comparison crates are development-only dependencies and are not included in consumer builds.

cargo bench --bench base64
cargo bench --bench murmur3
uv run --no-project python benchmarks/python_base64.py

To time only this implementation:

cargo bench --bench base64 -- hashcodecs
cargo bench --bench murmur3 -- hashcodecs
uv run --no-project python benchmarks/python_base64.py --hashcodecs-only

Base64: Rust

MurmurHash3: Rust

Base64: Python

Python decoding uses validate=True, and hashcodecs passes bytes directly into Rust without an input copy.

SIMD References

The AVX2 block structure uses the 24-byte encode and 32-byte decode arrangement described in Faster Base64 Encoding and Decoding using AVX2 Instructions. Base64 Turbo is included as a Rust comparator and is a useful direction for a future AVX-512 backend. This crate currently retains its Rust 1.85 MSRV, so its production runtime dispatch remains AVX2, SSSE3, and scalar.