YACE

Reversible text encoding with random substitution tables, layered re-encoding, noise chunks, and chunk shuffling.

YACE (Yet Another Custom Encoder) turns text into tokens made of random-looking letters and digits. Decoding a token needs the substitution tables that produced it, which you keep in a key store. It ships as a Python library and a command-line tool with no third-party dependencies.

This project is not affiliated with the yace package on PyPI ("Yet Another Code Emitter") or its yace-* companions, nor with YACE, the Prometheus CloudWatch exporter.

Warning: YACE obfuscates; it does not encrypt. At its core it is a substitution code with extra noise. Frequency analysis, or a couple of tokens whose text an attacker knows, can break it. Do not use it to protect passwords, personal data, API keys, or anything else that must stay secret. Use a vetted library such as cryptography for that.

Contents

• Why YACE
• How the encoder works
• Installation
• Quick start
• Command line
• API reference
• The key store
• Limits and edge cases
• Security
• Migrating from the original scripts
• Development
• Contributing
• License

Why YACE

Common encodings such as base64 or hex are public mappings, so anyone can reverse them. Decoding a YACE token needs the tables in your key store, and encoding the same text twice usually gives two different tokens.

You can use YACE to:

• keep text from being read at a glance, when it is not actually secret (puzzle or game content, test fixtures, or messages for people you share tables with);
• produce plain ASCII output: a short header line followed by a body of letters and digits only;
• learn how substitution, homophones, nulls, and transposition combine in classical ciphers.

How the encoder works

Tables, families, and the key store

A table maps each of the 100 characters in Python's string.printable to a random code of length ASCII letters and digits:

• every character except the space gets one code;
• the space gets 10 codes, and the encoder picks one at random for each space;
• each table also holds 26 bin codes: noise chunks that the encoder inserts and the decoder drops.

Every code in a table is unique, and no code repeats a character.

Tables are named <family>-<length>:<layer>, for example team-3:1. Encoding with family team, code length 3, and 2 iterations uses the tables team-3:1 and team-3:2. Missing tables are generated on first use and saved in the key store, a JSON file. The key store is the key: decoding needs the tables that encoded a token.

Encoding, step by step

Encoding "Hi Bob" with family demo, code length 2, and 1 iteration, using a seeded random generator so the example is reproducible:

1. Substitute. Each character is replaced by its code from demo-2:1 (H→lN, i→bx, B→bJ, o→La, b→4R). The space becomes ni, one of its 10 codes chosen at random.

   lN bx ni bJ La 4R                 6 chunks of 2 characters
   

2. Repeat for each iteration. With more iterations, layer k encodes the output of layer k−1 with table demo-2:k. Codes consist of letters and digits, which every table covers, so each layer multiplies the length by the code length.

3. Insert bins. ⌈chunks / 4⌉ random bin codes from the last table go into random chunk positions. Here ⌈6 / 4⌉ = 2, and the bin code J3 was picked twice:

   J3 lN bx ni bJ J3 La 4R           8 chunks
   

4. Swap chunks. For x = L, 2L, 3L, … (L is the code length) while x < n (the number of chunks), chunk x−1 swaps with chunk n−x. Here L = 2 and n = 8, so the pairs are (1, 6), (3, 4) and (5, 2):

   J3 La J3 bJ ni bx lN 4R
   

5. Add the header. The token is <family>-<length>:<iterations>, a newline, and the chunks:

   demo-2:1
   J3LaJ3bJnibxlN4R
   

Decoding reverses these steps. It reads the header, loads tables demo-2:1 … demo-2:I, undoes the swap (the swap is its own inverse), drops the bins, and maps codes back to characters from the last layer down to the first.

Token size

For a text of n characters, code length L, and I iterations:

• data chunks: m = n · L^I−1
• bins: ⌈m / 4⌉
• token body: L · (m + ⌈m / 4⌉) characters, plus the header line

With the defaults (L = 2, I = 3), "Hello, World!" (13 characters) becomes a 130-character body: about 10 times the input.

Design decisions

• Letters and digits only. The token body is safe to copy, paste, and embed, and each layer's output is valid input for the next layer.
• Unique codes. No two entries in a generated table share a code, so decoding is never ambiguous. (The original generator could produce shared codes; see Migrating.)
• Several codes for the space. The most common character in most text has no single code, so equal texts usually give different tokens.
• A fixed number of bins. The bin count depends only on the data length. The decoder uses this to check a token's length and to tell how many bins it holds.
• A self-inverse swap. The same function shuffles and unshuffles chunks. It uses no key, so it hides nothing from someone who knows the format.
• System randomness. Tables, space codes, and bins come from the operating system's cryptographic random source (secrets.SystemRandom), so tables are not predictable from a seed. This does not make the scheme secure (see Security).

Installation

python -m pip install yace-codec

On Windows, py -m pip install yace-codec works too.

To install the latest code from GitHub:

python -m pip install "git+https://github.com/thisismudith/YACE.git"

YACE needs Python 3.9 or newer and has no third-party dependencies.

Quick start

import yace_codec as yace

token = yace.encode("Hello, World!")
print(token)
print(yace.decode(token))

Example output (tables and tokens are random, so yours will differ):

default-2:3
IjC5wSwsIjMkC5wKIvpGZw7qxjed1zIj7qWlxjhp1zQcnNUNw4OrWlMkxj3DOmGaqSGDZwsGg17qsGYxSAwsg1QcwspGYxOrwSV9SApGwSZwxj7q1zao1zYxUNV9VGD4wG
Hello, World!

encode() and decode() use the default key store and the defaults name="default", length=2, iterations=3.

Choosing a key store, family, and settings

from yace_codec import Codec, KeyStore

codec = Codec(KeyStore("my-keystore.json"))
token = codec.encode("Meet me at 7pm.\nBring the blue folder!", "team", length=3, iterations=2)
print(token)
print(codec.decode(token))

team-3:2
rhvnH11s0X1f3cQ5VuX1fEh8cU0Y7KHJNX1fMajU1S2GYLrqvQMx6VC82uRbiRafm2X1fcpNY7K9HlihQNxSQbHu6nA6ljeU3cQMajUhZAER9HliRaHJNvQMdoC6GiQbHAzW5VuyLoPjYoFjqRaUhZX1frWSuRb104CDtcU0MajjeUmdwdV0PjYhWOA1xcU0IpNeamQbdIpNAERmdwU7yc5y6cHrhvNxSNr6Nr65VuyFD6GijLYvi0Wu8DJucpN104iRaU1SoFjfm2mTprWSdV03cQY7K6cHLrqvQMWu8N2xjLYIpNNxS2GYrZcMajMQZU1SQbHX1f104Y7KNxSHJN6cHEh8U1SROLQbH3cQNxS1W26GiNxS2GYdV0MajeamoFjSglCNS1043cQY7K3cQIpN6GiY7Kc5IU1SU7FX1fDJu
Meet me at 7pm.
Bring the blue folder!

A Codec reuses one key store, which is faster than the module-level functions when you encode many values.

Reading a token's header

header = yace.parse_header(token)
print(header)
print(header.family, header.length, header.iterations)
print(header.table_names)

team-3:2
team 3 2
('team-3:1', 'team-3:2')

Same text, different tokens

a = codec.encode("same text", "team", length=3, iterations=2)
b = codec.encode("same text", "team", length=3, iterations=2)
print(a == b, codec.decode(a) == codec.decode(b) == "same text")

False True

Empty and very short texts are the exception: they have so few possible tokens that the same token can come out twice.

Sharing tables

To let someone decode your tokens, export the family and have them import it. Only that family is shared; your other families stay private.

tables = codec.keystore.export_family("team")

colleague = Codec(KeyStore("colleague-keystore.json"))
print(colleague.keystore.import_tables(tables, family="team"))
print(colleague.decode(token))

ImportResult(added=('team-3:1', 'team-3:2'), unchanged=(), replaced=())
Meet me at 7pm.
Bring the blue folder!

Between machines, use yace-codec tables export and tables import (see Command line). On POSIX systems the export command creates the file with owner-only permissions.

When you import tables from someone else, import only the family you expect (family= in Python, --family NAME on the command line). A file could otherwise add tables for a family you use, such as default, and its sender could then decode your later tokens.

Unicode text and binary data

encode() accepts the 100 characters of string.printable. For any other text, or for binary data, use encode_bytes(). It base64-encodes the bytes first, which makes tokens about a third longer.

try:
    codec.encode("naïve café")
except yace.UnsupportedCharacterError as exc:
    print(exc.characters)

data = codec.encode_bytes("naïve café".encode("utf-8"), "team", length=3, iterations=1)
print(codec.decode_bytes(data).decode("utf-8"))

('é', 'ï')
naïve café

Handling errors

try:
    codec.decode("unknown-2:1\nAbCd")
except yace.TableNotFoundError as exc:
    print("missing:", exc.table_name)

try:
    codec.decode(token[:-1])
except yace.DecodingError as exc:
    print("corrupted:", exc)

missing: unknown-2:1
corrupted: layer 2 has 428 characters, which is not a multiple of the code length 3; the token is truncated or corrupted

Reproducible output in tests

Pass a seeded random.Random to get the same tables and tokens on every run. Use this only in tests: seeded tables are predictable.

import random

def make_token():
    seeded = Codec(KeyStore.in_memory(), rng=random.Random(42))
    return seeded.encode("reproducible", "test", length=2, iterations=1)

assert make_token() == make_token()

More runnable scripts are in examples/.

Command line

Installing the package adds the yace-codec command (python -m yace_codec works too). Input comes from a file argument, --text, or standard input. Output goes to -o PATH or standard output.

$ yace-codec encode --text "Hello, World!"
default-2:3
M0nsak8qa3KdTOLRKdKtenPBM0KMMtRfRfi6KMM0KfiKluO8O87d7dVW7dSzDLFp7dnsLRa2gL8qKfnsluMGGZ7dpSPBi6M0i4PBM0KMptRfWDuDFpKfpSHlaB8qiKi6kO

$ yace-codec encode --text "Hello, World!" | yace-codec decode
Hello, World!

Encode a file with your own family and settings, then decode it:

$ yace-codec encode notes.txt -o notes.yace --name groceries --length 3 --iterations 2
$ yace-codec decode notes.yace
Shopping list:
- milk
- 12 eggs

Encode any file byte for byte with --binary (use it again when decoding):

$ yace-codec encode --binary photo.jpg -o photo.yace
$ yace-codec decode --binary photo.yace -o photo-copy.jpg

Inspect and share tables:

$ yace-codec tables list
NAME       LENGTH  LAYERS
default    2       1,2,3
files      2       1
groceries  3       1,2

$ yace-codec tables export groceries -o groceries.json
$ yace-codec tables import groceries.json --family groceries --keystore colleague.json
added: groceries-3:1
added: groceries-3:2
2 added, 0 unchanged, 0 replaced (colleague.json)

$ yace-codec tables path
C:\Users\you\AppData\Roaming\yace-codec\keystore.json

tables import needs --family NAME, which imports only that family's tables, or --all-families for a file you trust completely. A file could otherwise add tables to a family you use, such as default, and its sender could then decode your later tokens:

$ yace-codec tables import groceries.json
yace-codec: error: groceries.json holds tables for: groceries. Name the family you expect with --family NAME, or import everything with --all-families

Text mode reads and writes UTF-8 and keeps line endings unchanged, so decode returns exactly the bytes that were encoded. Token and JSON files saved by Windows PowerShell (UTF-16, or UTF-8 with a byte order mark) are accepted. Errors print yace-codec: error: … and exit with status 1. Invalid arguments exit with status 2.

API reference

Everything below is importable from yace_codec.

Functions

keystore accepts a KeyStore, a path to a key store file, or None for the default key store.

Codec(keystore=None, *, rng=None)

Encodes and decodes with one key store. It has the same encode, decode, encode_bytes, and decode_bytes methods as the functions above (without the keystore argument), and a keystore attribute. rng replaces the system random source with a random.Random, for reproducible tests only.

KeyStore(path)

A key store file. A relative path is resolved against the current directory when the KeyStore is created. The file is created on the first write.

import_tables validates every table before it writes anything, and it writes nothing when every table is already present. It raises KeyStoreError if a table already exists with different codes, unless overwrite=True.

Data types

• Header(family, length, iterations): a named tuple. str(header) gives the header line, and header.table_names lists the tables needed for decoding.
• Table: an immutable, validated table with name, length, codes (a read-only mapping from character to code), space_codes, bin_codes, and characters (what it can encode), plus to_dict(), Table.from_dict(name, data), and Table.generate(name, rng).

Exceptions

YaceError
├── InvalidNameError           invalid family name (also a ValueError)
├── EncodingError              invalid settings (also a ValueError)
│   └── UnsupportedCharacterError   .characters, .table_name
├── DecodingError              malformed or corrupted token, or wrong tables (also a ValueError)
├── TableNotFoundError         a needed table is missing; .table_name (also a LookupError)
└── KeyStoreError              unreadable, invalid, or unwritable key store

Constants

DEFAULT_FAMILY ("default"), DEFAULT_LENGTH (2), DEFAULT_ITERATIONS (3), MIN_LENGTH (2), MAX_LENGTH (62), MAX_ITERATIONS (32), MAX_ENCODED_CHARS (2²⁵), SUPPORTED_CHARACTERS, ENV_VAR ("YACE_CODEC_KEYSTORE"), and __version__.

The key store

The default key store is:

Set the YACE_CODEC_KEYSTORE environment variable, pass --keystore to the command, or pass a path or KeyStore to the API to use another file.

The file is a JSON object that maps table names to tables, in the same format as the original scripts' encodings.json:

{"team-3:1": {"!": "B1x", "a": "yI4", "...": "...", " ": ["6nM", "..."], "bin": ["EbC", "..."]}}

How the key store behaves:

• Existing tables are never changed. New tables are only added, except through import_tables(..., overwrite=True).
• Writes are atomic. Data goes to a temporary file that then replaces the key store, so a crash cannot leave a half-written file. A symbolic link is followed, so the file it points to is updated and the link stays. A key store with more than one hard link is refused instead, because replacing it would split the copies.
• Permissions are kept. New key store files get mode 0600, and a new yace-codec directory gets mode 0700, on POSIX. An existing key store keeps its access control list on Windows, and its mode on POSIX, plus its owner and group where the system allows it.
• Concurrent writers are safe. Writes are serialised with a keystore.json.lock file next to the key store, so parallel processes never lose each other's tables. Reading, and encoding with tables that already exist, need no write access.
• Damaged files are left alone. If the file is not valid JSON, you get a KeyStoreError and the file is not overwritten. Entries the package does not understand are kept.

Back up your key store. If you lose it, every token encoded with its tables becomes undecodable. Anyone who has it can decode your tokens.

Limits and edge cases

• Characters. encode() supports the 100 characters of string.printable: ASCII letters, digits, punctuation, space, \t, \n, \r, \x0b, and \x0c. It raises UnsupportedCharacterError before creating any table if the text contains anything else. Use encode_bytes() for other text. Tables from the original scripts do not cover \t, \r, \x0b, or \x0c.
• Settings. length must be 2–62 and iterations 1–32. Encoding is refused if len(text) × length ** iterations exceeds 2²⁵ (about 33 million characters before bins; with bins, tokens reach about 42 million characters).
• Size. Tokens grow by length ** iterations, plus 25% for bins (see Token size). Keep iterations small for long texts.
• Memory and speed. Encoding and decoding hold the whole input and token in memory and are written in pure Python. They need roughly 25–40 bytes of memory per token character, so tokens near the size limit need 1–2 GB. YACE is meant for strings and small to medium files, not streams.
• Empty input. Encoding "" gives a header-only token, which decodes to "".
• Whitespace. Decoding ignores whitespace in a token, so wrapped lines, CRLF line endings, and leading blank lines are fine.
• Corruption. Decoding raises DecodingError when a chunk is not a valid code, when the length does not fit the code length, or when the number of noise chunks does not match the token length. These checks catch many truncations, but not all: removing or swapping data chunks can go unnoticed. Tokens have no cryptographic integrity check.
• Bytes tokens. decode_bytes() raises DecodingError if the decoded text is not base64. Short texts can happen to be valid base64, so keep track of which function made a token.
• Family names. Encoding needs a family name of 1–64 ASCII letters or underscores, including for existing families from the original scripts. Digits, -, and : are reserved for table names. Decoding accepts any family name the original scripts allowed, such as my cat or café, except a name that starts with a space.

Security

• YACE is a randomised substitution code with a keyless, public chunk shuffle. Within a family, every character except the space always has the same code. So frequency analysis over enough tokens recovers the text, and a couple of tokens whose text an attacker knows can be enough to decode other tokens.
• The header shows the family name, code length, and iteration count. The token length reveals the exact length of the text (for encode_bytes, of its base64 form).
• Tokens have no integrity protection or authentication.
• The key store is the key. Store it like a credential, share single families with tables export instead of sharing the whole file, and import only the family you expect from files other people send you.
• Use real encryption (for example AES-GCM or Fernet from cryptography) whenever confidentiality matters.

To report a bug, including a security issue in the code, please open an issue.

Migrating from the original scripts

Before version 0.1.0, YACE was a set of scripts in encoder/ that stored tables in encoder/encodings.json. The package keeps the same algorithm, table format, and token format.

• Keep your old tables. This release no longer includes encoder/encodings.json. If you still have tokens made with the original scripts, keep your copy of that file: it is the key store those tokens need.
• Old tokens decode with the package. Use your old encodings.json as the key store (--keystore path/to/encodings.json, or KeyStore("path/to/encodings.json")), or import its tables with yace-codec tables import path/to/encodings.json --all-families.
• New tokens decode with the original scripts when the package generated the tables. With tables from the original generator, the original decoder can return wrong text when a token contains a character whose code that table also uses as a bin. About half of the original generator's length-2 tables have such a code.

Behaviour changes:

• Errors raise exceptions instead of printing a message and calling exit().
• Characters a table cannot encode raise UnsupportedCharacterError. Previously they were copied into the token unencoded, which corrupted it.
• Empty text works, a code length of 1 is rejected (it used to hang), and family names with repeated letters such as hello are accepted (the old check rejected them by mistake).
• Encoding needs a family name of ASCII letters and underscores, even for existing families. Families with other names, such as my cat, can still be decoded.
• Without a name, the family default is used instead of random letters.
• New tables cover tab, carriage return, vertical tab, and form feed, and never share codes.
• The original generator sometimes gave a space or a bin the same code as another entry, which could make the original decoder return garbled text. The package never emits a shared space or bin code. (A character keeps its code even when a legacy table also uses that code as a bin.) When it decodes old tokens, it uses the token length to tell how many bins they hold, and the fact that inner layers can only contain letters and digits. This recovers many tokens the original decoder could not. Some remain ambiguous: for example, a space that was encoded with a code shared by a character.
• randomizer_data.json, errors.log, and the unfinished file analyser were removed. Use yace-codec encode FILE (or --binary) to encode files.

Development

git clone https://github.com/thisismudith/YACE.git
cd YACE
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
python -m pip install -e .
python examples/quickstart.py

To build the wheel and source distribution into dist/, run python -m pip install build and then python -m build.

Contributing

Issues and pull requests are welcome at https://github.com/thisismudith/YACE.

1. Open an issue first for larger changes.
2. Keep the token and key store formats backward compatible: tokens made by earlier versions, and by the original scripts, must keep decoding.
3. Run the scripts in examples/ to check your change.

License

MIT. See LICENSE.

Links

• Source: https://github.com/thisismudith/YACE
• Issues: https://github.com/thisismudith/YACE/issues
• PyPI: https://pypi.org/project/yace-codec/
• Changelog: https://github.com/thisismudith/YACE/blob/main/CHANGELOG.md

Author

Mudith Daga (@thisismudith)