z85base91

A binary-to-text encoding library. It provides three codecs: Base91, Z85B, and Z85P. Each codec turns binary data into printable ASCII text for transport over text-oriented channels, with a fast C implementation and a pure-Python fallback, so the library works on any platform. The HiveMind mesh uses it to pack binary payloads for text-mode transports.

Why z85base91?

Base64 expands data by 33% (1.33x). Base91 expands data by only 23% (1.23x), and stays more compact than base32 (62% overhead). The Z85 variants sit between these: Z85B expands data by 1.25x, and Z85P by 1.28x. Both add character-safety guarantees useful for sending encrypted frames over websockets and other text transports.

Comparison

Codec	Expansion	Use Case
Base91	1.23x (smallest)	Maximum compactness, printable ASCII
Z85B	1.25x	Z85 padding scheme, per-group framing
Z85P	1.28x	Z85 padding scheme, prepended size byte
base64	1.35x	Standard, less compact
base32	1.62x	Case-insensitive, largest

Installation

pip install z85base91

The package bundles precompiled C extensions for x86_64, i386, and aarch64. If the C library fails to load, the package falls back to pure Python and logs a warning.

Quick Start

from z85base91 import B91, Z85B, Z85P

# Base91 (most compact)
data = b"Hello, World!"
encoded = B91.encode(data)        # b'>OwJh>}AQ;r@@Y?F'
decoded = B91.decode(encoded)     # b'Hello, World!'

# Z85B (Z85 with independent groups)
encoded = Z85B.encode(data)
decoded = Z85B.decode(encoded)

# Z85P (Z85 with prepended padding byte)
encoded = Z85P.encode(data)
decoded = Z85P.decode(encoded)

All three codecs accept str or bytes input and return bytes.

Public API

Base91 (B91)

from z85base91 import B91

Encoding

B91.encode(data: Union[str, bytes], encoding: str = "utf-8") -> bytes

Encodes binary data with Base91. Base91 uses 91 printable ASCII characters (A-Z, a-z, 0-9, plus 27 symbols) and expands data by 1.23x.

Arguments:

• data: Input as str (encoded to UTF-8) or raw bytes.
• encoding: Character encoding to use if data is str. Default: "utf-8".

Returns: Base91-encoded bytes.

Example:

B91.encode(b"test")           # b'fPNKd'
B91.encode("test")            # b'fPNKd' (auto UTF-8)
B91.encode(b"\x00\x01\x02")   # b':CQA'

Decoding

B91.decode(encoded_data: Union[str, bytes], encoding: str = "utf-8") -> bytes

Decodes Base91-encoded input back to raw bytes.

Arguments:

• encoded_data: Base91 string or bytes.
• encoding: Character encoding to use for string conversion. Default: "utf-8".

Returns: Decoded bytes.

Raises: ValueError if the input has a character outside the 91-character alphabet.

Example:

B91.decode(b'>OwJh>}AQ;r@@Y?F')  # b'Hello, World!'
B91.decode('>OwJh>}AQ;r@@Y?F')   # b'Hello, World!' (str input)

---

Z85B (Z85B)

from z85base91 import Z85B

Encoding

Z85B.encode(data: Union[str, bytes]) -> bytes

Encodes binary data with Z85B, a Z85 variant that processes 4-byte chunks independently. It uses the 85-character Z85 alphabet: 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ.-:+=^!/*?&<>()[]{}@%$#. Expansion is about 1.25x.

Arguments:

• data: Input as str (encoded to UTF-8) or raw bytes.

Returns: Z85B-encoded bytes.

Example:

Z85B.encode(b"Hello")         # b'NV0&yq1' (5 bytes -> 7 bytes)
Z85B.encode("Hello")          # b'NV0&yq1' (str auto UTF-8)

Decoding

Z85B.decode(encoded_data: Union[str, bytes]) -> bytes

Decodes Z85B-encoded input.

Arguments:

• encoded_data: Z85B-encoded string or bytes.

Returns: Decoded bytes.

Raises: ValueError if the input has a character outside the Z85 alphabet.

Example:

original = b"Hello, World!"
encoded = Z85B.encode(original)
decoded = Z85B.decode(encoded)
assert decoded == original

---

Z85P (Z85P)

from z85base91 import Z85P

Encoding

Z85P.encode(data: Union[str, bytes]) -> bytes

Encodes binary data with Z85P, a Z85 variant that prepends an explicit padding byte. The first byte of the output states how many padding bytes were added (0-3), so the decoder can strip them. Z85P uses the same 85-character Z85 alphabet as Z85B. Expansion is about 1.28x.

Arguments:

• data: Input as str (encoded to UTF-8) or raw bytes.

Returns: Z85P-encoded bytes (first byte is the padding indicator, 0-3).

Example:

Z85P.encode(b"A")      # b'\x03k(Z(+' (1 byte + 3 padding = 4, encodes to 5)
Z85P.encode(b"AB")     # b'\x02k%+LK' (2 bytes + 2 padding = 4)
Z85P.encode(b"ABCD")   # b'\x00k%^}b' (4 bytes, no padding)

Decoding

Z85P.decode(encoded_data: Union[str, bytes]) -> bytes

Decodes Z85P-encoded input and strips the padding based on the first byte.

Arguments:

• encoded_data: Z85P-encoded string or bytes (must include the padding indicator byte).

Returns: Decoded bytes with padding removed.

Raises: ValueError if the input length is invalid or has a character outside the Z85 alphabet.

Example:

original = b"X"
encoded = Z85P.encode(original)
decoded = Z85P.decode(encoded)
assert decoded == original

---

Performance Notes

C vs. Python

The C implementations run substantially faster than the Python fallback (about 1.5x for decoding, up to 2x for encoding). The library selects the C path automatically. The Python fallback runs only if compilation failed or the C library failed to load.

The package ships a benchmark harness (z85base91/bench.py) that compares all codecs against stdlib base64/base32 across input sizes. It needs click, tabulate, pybase64, and hivemind-bus-client (for the pure-Python reference codecs), so install those before you run it:

pip install click tabulate pybase64 hivemind-bus-client
python -m z85base91.bench

Expansion ratios are deterministic: Base91 is about 1.23x, Z85B about 1.25x, and Z85P about 1.25x for aligned payloads (up to 1.28x for short, heavily padded inputs), against base64's 1.33x and base32's 1.6x. Absolute throughput depends on hardware. The C path is consistently faster than stdlib base64 for encoding, and faster than the pure Python fallback for both directions.

---

Architecture

The library has three layers:

1. Public API (z85base91/__init__.py): exports the B91, Z85B, and Z85P classes. At import time, each class tries to load its architecture-specific C library (.so) with ctypes.CDLL. If loading fails, the name is rebound to the pure-Python implementation.

2. C implementations (src/*.c, prebuilt as z85base91/lib*-{arch}.so):

   • libbase91-{x86_64,aarch64,i386}.so: B91 codec
   • libz85b-{x86_64,aarch64,i386}.so: Z85B codec
   • libz85p-{x86_64,aarch64,i386}.so: Z85P codec

3. Pure-Python fallbacks (z85base91/{b91,z85b,z85p}.py): self-contained reference implementations.

Data flow: normalize input to bytes, wrap it in a ctypes.c_ubyte array, call the C function, and read the output with ctypes.string_at.

---

Error Handling

Invalid input characters

Every decoder raises ValueError if the input has a character outside the allowed alphabet:

try:
    Z85P.decode("Hello€World")
except ValueError as e:
    print(f"Invalid character: {e}")

Missing C library

If the C library fails to load, the package logs a WARNING and uses the Python fallback:

WARNING - Z85P C library not available: Library load error. Falling back to pure Python implementation.

No exception is raised. Encoding and decoding still work, just slower.

---

HiveMind Integration

The HiveMind mesh uses this library to pack binary payloads (encrypted frames, keys, metadata) for transmission over websocket channels, where text-mode framing is required. The 23% overhead of Base91, against 33% for base64, saves bandwidth at scale.

Example: a 1 MB encrypted message expands to 1.23 MB with Base91, against 1.35 MB with base64, a saving of about 12 KB per message that adds up across thousands of mesh nodes.

---

Requirements

Runtime: Python 3.8+ (no external dependencies, ctypes is part of the standard library)

Build: python3-dev, swig (to compile the C extensions)

Tests: pytest~=7.1, pytest-cov~=4.1

---

Testing

pip install -e . --no-deps
pip install -r test/requirements.txt
pytest test/ -q

The test suite covers all codecs for:

• Round-trip encode/decode (identity)
• Edge cases (empty input, single byte, odd lengths)
• Invalid input handling
• Unicode string support
• Large payloads (1000+ bytes)

---

Related Projects

• hivemind-websocket-client: the HiveMind protocol client. Its encodings/ module is the origin of the Base91/Z85B/Z85P codecs this library ships as a standalone, C-accelerated package.
• hivemind-bus-client: provides the pure-Python reference codecs used by the benchmark harness.

License

Apache 2.0