xor-delta 1.0.0

Reversible adjacent XOR differencing transform algo

XOR ∆

A reversible adjacent XOR differencing transform algo

"To compress, perchance to save..."

Synopsis

xor-delta is a small experimental Python package that explores XOR-adjacent delta encoding as a preprocessing transform for compression.

It answers a very specific question:

Does XORing adjacent values reduce entropy in a way that helps real compressors?

• Short answer: maybe 🤷🏼‍♂️
• Long answer: Step into my office...

---

What is XOR-delta?

For Those in a Hurry...

The core transform used by this project is:

A_i^{(k+1)} = A_i^{(k)} \oplus A_{i+1}^{(k)} \quad \forall i

Where:

• $A^{(k)}$ is the original sequence at step $k$
• $\oplus$ denotes bitwise XOR
• One endpoint value (the anchor) is stored to make the transform reversible

How Does It Work?

Given a sequence of values:

v0, v1, v2, v3, ...

XOR-delta encoding stores:

• one anchor value (first or last)
• a list of XORs between adjacent values:

v0 ^ v1, v1 ^ v2, v2 ^ v3, ...

This transform is:

• lossless
• reversible
• cheap
• not compression by itself

It’s a preprocessing step you can feed into standard compressors like zlib, bz2, or lzma.

Installation

pip install xor-delta

Or

git clone https://GitHub.com/DJStompZone/xor_delta
cd xor_delta
pip install . # use `--with=dev` if you plan to run tests

Python API

Integer sequences

from xor_delta import xor_delta_encode_ints, xor_delta_decode_ints

data = [10, 11, 12, 13]

encoded = xor_delta_encode_ints(data)
decoded = xor_delta_decode_ints(encoded)

assert decoded == data

Byte sequences

from xor_delta import xor_delta_encode_bytes, xor_delta_decode_bytes

data = b"hello world"

anchor, diffs, side = xor_delta_encode_bytes(data)
restored = xor_delta_decode_bytes(anchor, diffs, side)

assert restored == data

CLI Benchmark Tool

xor-delta ships with a benchmarking CLI that compares compression before and after XOR-delta.

Run the default benchmark (Shakespeare)

xor-delta-bench

This automatically downloads Shakespeare from Project Gutenberg, caches it locally, and benchmarks:

• raw bytes
• XOR-adjacent bytes

using:

• zlib
• bz2
• lzma

corpus_cache/pg100.txt.<hash>
  RAW      raw=5,638,525  zlib=2,138,296 (0.379x)  bz2=1,586,908 (0.281x)  lzma=1,673,804 (0.297x)
  XOR      raw=5,638,525  zlib=2,546,436 (0.452x)  bz2=1,708,046 (0.303x)  lzma=1,890,440 (0.335x)
  xor-vs-raw  zlib +19.09%   bz2 +7.63%   lzma +12.94%

Interpretation:
XOR-delta made compression worse for English text across all tested compressors.

That’s the point — we measured it instead of guessing.

Benchmark your own files

xor-delta-bench myfile.bin
xor-delta-bench mydir/

Use a Gutenberg preset

xor-delta-bench --gutenberg shakespeare
# Feel free to send a PR if you want more presets <3

Or any URL

xor-delta-bench --gutenberg-url https://example.com/text.txt

Downloads are cached in corpus_cache/.

---

When does XOR-delta help?

XOR-adjacent transforms can help when:

• data has small local variation
• values are structured, not textual
• adjacent samples are correlated

Examples:

• counters
• timestamps
• some sensor streams
• monotonic-ish numeric data

It can hurt when:

• data is already high-entropy
• compressors already exploit structure better (text + LZ)
• XOR destroys symbol locality

Development

Run tests:

pytest
# Or if you're using Poetry
poetry run pytest

License

MIT

Credits

Created by DJ Stomp https://github.com/DJStompZone/xor_delta

Inspired by spectcow's original description of the algorithm, full credit for the core concept goes to them.