pybased 0.0.2

Library for creating arbitrary binary encodings. Includes variations on base32, base64, base85, and more.

pybased

Library for creating arbitrary binary encodings. Includes variations on base32, base64, base85, and more.

WARNING: Although these encodings do work end-to-end, they are not compatible with traditional implementations!

I am working on finding out why.

Encodings

Types

• Sliding: Random-access bitwise implementation, theoretically compatible with bytestrings of arbitrary size.
• BigInt: Converts bytes into large integers, then incrementally divides by the radix. Incompatible with large files. Suitable for hashes and shorter bytestrings.
• Biterator: Iterates through each byte from the input stream and adds it to a buffer, then extracts the required number of bits to convert to the other radix. Based roughly off of CPython's base64 encoder, but heavily modified to enable arbitrary conversions. WIP.

Available Encodings

Supported Encodings

ID	Type	Bits/Char	Chars/Byte	Alphabet
base32	Biterator	5	5	ABCDEFGHIJKLMNOPQRSTUVWXYZ234567
base32hex	Biterator	5	5	0123456789ABCDEFGHIJKLMNOPQRSTUV
base46	BigInt	N/A	N/A	ABCDEFGHJKMNPQRSTUVWXYZabcdefghjkmnpqrstuvwxyz
base62	BigInt	N/A	N/A	0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
base64	Biterator	6	3	ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/
base64b64	Biterator	6	3	./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
base64bash	Biterator	6	3	0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ@_
base64bcrypt	Biterator	6	3	./ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789
base64hqx	Biterator	6	3	!"#$%&'()*+,-012345689@ABCDEFGHIJKLMNPQRSTUVXYZ[`abcdefhijklmpqr
base64url	Biterator	6	3	ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_
base64uu	Biterator	6	3	!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_
base64xx	Biterator	6	3	+-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
base85	Biterator	7	7	ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!#$%&()*+-;<=>?@^_`{|}~
base94	BigInt	N/A	N/A	!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
crockford32	Biterator	5	5	0123456789ABCDEFGHJKMNPQRSTVWXYZ
geohash32	Biterator	5	5	0123456789bcdefghjkmnpqrstuvwxyz
nintendo32	Biterator	5	5	0123456789BCDFGHJKLMNPQRSTVWXYZ?
wordsafe32	Biterator	5	5	23456789CFGHJMPQRVWXcfghjmpqrvwx
zbase32	Biterator	5	5	ybndrfg8ejkmcpqxot1uwisza345h769

Status

Supported Encodings

Standard	Encoded	Decoded	Passed Test
base32	JBSWY3DPFQQHO33SNRSCC===	b'Hello, world!'	✔
base32hex	91IMOR3F5GG7ERRIDHI22===	b'Hello, world!'	✔
base46	GnNaagudEKYvFzFrZSS	b'Hello, world!'	✔
base62	1wJfrzvdbthTq5ANZB	b'Hello, world!'	✔
base64	SGVsbG8sIHdvcmxkIQ==	b'Hello, world!'	✔
base64b64	G4JgP4wg65RjQalY6E==	b'Hello, world!'	✔
base64bash	i6lIr6YI87tLsCNA8g==	b'Hello, world!'	✔
base64bcrypt	QETqZE6qGFbtakviGO==	b'Hello, world!'	✔
base64hqx	5'9XE'mX)(G[FQaN)3==	b'Hello, world!'	✔
base64url	SGVsbG8sIHdvcmxkIQ==	b'Hello, world!'	✔
base64uu	2&5L;&\\L('=O<FQD(0	b'Hello, world!'	✔
base64xx	G4JgP4wg65RjQalY6E==	b'Hello, world!'	✔
base85	ERR	ERR	❌
base94	/P\|?l:+>Nq\\sr<+r	b'Hello, world!'	✔
crockford32	91JPRV3F5GG7EVVJDHJ22	b'Hello, world!'	✔
geohash32	91kqsv3g5hh7fvvkejk22	b'Hello, world!'	✔
nintendo32	91LQSW3H5JJ7GWWLFKL22	b'Hello, world!'	✔
wordsafe32	F3Wgjq5Q7RR9PqqWMVW44	b'Hello, world!'	✔
zbase32	jb1sa5dxfoo8q551pt1nn	b'Hello, world!'	✔

Errors:

base85: c = 99, len = 85, bpc = 7

Getting started

$ pip install pybased

Doing stuff

# Lets's assume we want to use the Crockford32 encoding scheme.
from based.standards.base32 import crockford32

# And let's assume the variable data has what we want to encode.
data: bytes = ...

# Encode to string.
encoded: str = crockford32.encode_bytes(data)

# ...

# Decode the string back to bytes.
data: bytes = crockford32.decode_bytes(encoded)

based Command-Line Tool

$ based --help

usage: based [-h] {dump,encode,decode} ...

positional arguments:
  {dump,encode,decode}

optional arguments:
  -h, --help            show this help message and exit

NOTE: The based CLI tool is currently only useful for testing, and is under very active development.

Encode string to Base94

$ based encode --standard=base94 --input-string 'Hello, world!'

/P|?l:+>Nq\sr<+r

Encode string to Base94 and output JSON

$ based encode --standard=base94 --input-string 'Hello, world!' --output-format json

{"encoded": "/P|?l:+>Nq\\sr<+r"}

Decode data from Base94

$ based decode --standard=base94 --input-string '/P|?l:+>Nq\sr<+r'

>>> input chars: 16B
Bytes representation: b'Hello, world!'
Hex representation: 48656c6c6f2c20776f726c6421

Various Output Encoding Formats

$ based decode --standard=base94 --input-string '/P|?l:+>Nq\sr<+r' --output-format json

>>> input chars: 16B
{"b64": "SGVsbG8sIHdvcmxkIQ==", "utf-8": "Hello, world!"}

$ based decode --standard=base94 --input-string '/P|?l:+>Nq\sr<+r' --output-format yaml

>>> input chars: 16B
encoded: !!binary |
  SGVsbG8sIHdvcmxkIQ==