hashcodecs

SIMD-accelerated Base64, MurmurHash3, and XXH3 for Python and Rust.

Move byte-heavy work into Rust without changing your Python inputs. hashcodecs accepts bytes, bytearray, and memoryview, selects the highest-priority supported SIMD backend, and exposes batch and reusable-buffer APIs.

Features

• Base64 encode and decode with standard, URL-safe, padded, unpadded, wrapped, and canonical modes.
• MurmurHash3 x86-32, x86-128, and x64-128 with one-shot and incremental APIs.
• Bit-for-bit compatible XXH3-64 and XXH3-128 with prepared seeds and allocating and allocation-free native batch APIs.
• Caller-managed *_into outputs for allocation-sensitive workloads.
• Runtime dispatch across AVX-512, AVX2, SSE4.1, SSSE3, NEON, and scalar implementations where applicable.
• Direct CPython buffer handling for bytes, bytearray, and memoryview inputs.
• Install wheels for CPython 3.10 through 3.15 and free-threaded CPython 3.14t and 3.15t on Linux, macOS, and Windows.

Installation

pip3 install hashcodecs

Python

The Base64 module follows familiar Python conventions while adding explicit padding, canonical validation, batch, and reusable-buffer operations.

import hashcodecs.base64 as base64
from hashcodecs import murmur3_32, xxh3_64, xxh3_128_batch

assert base64.b64encode(b'hello') == b'aGVsbG8='
assert base64.b64decode(b'aGVsbG8=') == b'hello'
assert base64.urlsafe_b64encode(b'hello', padded=False) == b'aGVsbG8'

assert murmur3_32(b'hello') == 0x248BFA47
assert xxh3_64(b'') == 0x2D06800538D394C2
assert xxh3_128_batch([b'hello', b'world']) == [
    0xB5E9C1AD071B3E7FC779CFAA5E523818,
    0xFA0D38A9B38280D0891E4985BDB2583E,
]

Reusable outputs

*_into functions write into caller-managed bytearray storage and return the number of bytes written. XXH3 batch outputs are packed little-endian digests.

import hashcodecs.base64 as base64
from hashcodecs import xxh3_64_batch_into

payload = b'hello'
encoded = bytearray(4 * ((len(payload) + 2) // 3))
encoded_len = base64.b64encode_into(payload, encoded)
assert encoded[:encoded_len] == b'aGVsbG8='

hashes = bytearray(2 * 8)
written = xxh3_64_batch_into([b'hello', b'world'], hashes, seed=42)
assert written == 16

Incremental hashing

from hashcodecs import murmur3_x64_128

hasher = murmur3_x64_128(seed=42)
hasher.update(b'hello')
snapshot = hasher.copy()
hasher.update(b' world')

assert snapshot.hexdigest() == snapshot.digest().hex()
assert hasher.digest() != snapshot.digest()

Rust

The Rust API exposes the same core algorithms without the Python binding layer and requires Rust 1.94 or newer.

cargo add hashcodecs

let encoded = hashcodecs::base64::b64encode(b"hello");
assert_eq!(encoded, "aGVsbG8=");

let mut output = [0_u8; 8];
let written = hashcodecs::base64::b64encode_into(b"hello", &mut output).unwrap();
assert_eq!(&output[..written], b"aGVsbG8=");

assert_eq!(
    hashcodecs::murmur3::murmur3_x86_32(b"hello", 0),
    0x248b_fa47
);
assert_eq!(
    hashcodecs::xxhash::xxh3_64(b"", 0),
    0x2d06_8005_38d3_94c2
);

let seeded_input = vec![7; 1024];
let prepared = hashcodecs::xxhash::PreparedXxh3::new(42);
assert_eq!(
    prepared.hash_64(&seeded_input),
    hashcodecs::xxhash::xxh3_64(&seeded_input, 42)
);

let inputs: &[&[u8]] = &[b"hello", b"world"];
assert_eq!(
    prepared.hash_64_batch(inputs),
    hashcodecs::xxhash::xxh3_64_batch(inputs, 42)
);
let mut hashes = [0_u64; 2];
let mut index = 0;
hashcodecs::xxhash::xxh3_64_batch_for_each(inputs, 0, |hash| {
    hashes[index] = hash;
    index += 1;
});
assert_eq!(index, inputs.len());

Architecture

The Rust core owns algorithm behavior and SIMD dispatch. The CPython layer handles argument parsing, buffers, reusable outputs, and GIL decisions. Root-level Python modules provide typed exports without per-call wrappers.

Each Rust algorithm exposes a small public module. Base64 groups internals by encode and decode operation and places ISA kernels such as encode/avx2.rs and decode/ssse3.rs under their operation. MurmurHash3 groups code by canonical variant. XXH3 uses processing-stage modules. Long-input ISA kernels are under xxhash/long_inputs/.

See docs/ARCHITECTURE.md for the module layout, dispatch model, algorithm data flows, CPython boundary, and safety invariants.

Benchmarks

Run the suite on Windows 10 x64 with an Intel Core Ultra 7 265K. Pin one logical CPU and run each case in one thread. Collect 50 Rust samples and 15 Python samples. Higher throughput wins. Use free-threaded CPython 3.15.0 with the GIL disabled for all Python charts.

Base64: Rust

MurmurHash3: Rust

XXH3: Rust

Link hashcodecs with xxHash 0.8.3 through xxhash-c-sys. Build the C baseline with AVX2. Batch cases pass 32 equal-size inputs and include result-vector allocation.

Base64: Python

Pass bytes to Rust without an input copy. Python decoding uses validate=True.

MurmurHash3: Python

XXH3: Python

Pass 32 equal-size inputs to each batch case. Compare one native hashcodecs call with a loop over the upstream xxhash extension.

Read the focused cases, commands, and values in BENCHMARK.md. Read raw chart values in docs/benchmarks/results.csv.

Reproduce

Comparison crates and Python packages are development-only dependencies and are not included in consumer builds.

cargo bench --manifest-path benches/Cargo.toml --bench base64
cargo bench --manifest-path benches/Cargo.toml --bench murmur3
cargo bench --manifest-path benches/Cargo.toml --bench xxhash
cargo bench --manifest-path benches/Cargo.toml --bench crossover

uv sync --python 3.15.0t --frozen --group benchmark --no-install-project
uv run --python 3.15.0t --frozen --no-sync python tools/install_local_wheel.py
uv run --python 3.15.0t --frozen --no-sync python benchmarks/python_base64.py
uv run --python 3.15.0t --frozen --no-sync python benchmarks/python_base64_batch.py
uv run --python 3.15.0t --frozen --no-sync python benchmarks/python_calls.py
uv run --python 3.15.0t --frozen --no-sync python benchmarks/python_murmur3.py
uv run --python 3.15.0t --frozen --no-sync python benchmarks/python_xxhash.py

The Python benchmarks expose focused modes such as --into, --lenient, --bytearray-input, --memoryview-input, --sliced-memoryview-input, --buffer-inputs, --incremental, --large, and --hashcodecs-only. All scripts also accept --samples and --minimum-sample-seconds; use --help on a benchmark script for its supported modes and defaults.

For the same-ISA Windows XXH3 comparison shown above, rebuild the C baseline with:

$env:CFLAGS='/O2 /arch:AVX2'
cargo clean -p xxhash-c-sys
cargo bench --manifest-path benches/Cargo.toml --bench xxhash

Performance snapshot

On the benchmark host with CPython 3.15.0t, hashcodecs.xxh3_64 processes a 1 MiB input at 82.39 GiB/s. The Base64 batch API reaches 5.46 GiB/s for encode and 5.11 GiB/s for decode with 256 B items in batches of 64. Each run pins one logical CPU and uses 15 samples with a 0.2-second minimum per sample. Read the benchmark details and raw comparison results.

Development

Build the Python wheel and source distribution:

uv build --sdist --wheel

For a wheel alone, use uv build --wheel.

Run the primary local checks:

just check

just check builds and reinstalls the current wheel before running Python tests. Run just full-check before committing to also check release builds, Rust core coverage, and the extracted source distribution.

Optimized paths are also checked with differential fuzzing, Kani, strict-provenance Miri, AddressSanitizer, and MemorySanitizer in CI.

Compatibility

Version 1.x keeps the documented Python API stable under the compatibility policy. Release wheels target CPython 3.10 through 3.15 on manylinux x86-64, macOS 11+ ARM64, and Windows x86-64. CPython 3.14t and 3.15t receive free-threaded wheels on the same platforms.

The Rust crate and Python package share one release version. The documented Python API follows the compatibility policy below. See Security Policy for vulnerability reporting.

References

The Base64 SIMD implementation follows the approach described in Faster Base64 Encoding and Decoding using AVX2 Instructions, extended with runtime-selected AVX-512 VBMI and AArch64 NEON backends.

Citation

If you use hashcodecs, cite CITATION.cff.