lzstring-codec

lz-string for Python, held byte-for-byte to the JavaScript original.

Installed as lzstring-codec, imported as lz_string. The obvious name is not available: PyPI refuses it as too similar to lzstring, which is one of the two packages this exists to replace.

It exists because the package that is on PyPI gets the format wrong in ways that quietly corrupt real save files. It was written for saveeditor.online, which reads a few thousand lz-string saves a week and cannot afford to hand any of them back altered.

import lz_string as lz

lz.compress_to_base64('{"gold":9000}')          # 'N4Ig5g9gNgJiBcBOADKgvkA='
lz.decompress_from_base64(_)                    # '{"gold":9000}'

Why

lzstring 1.0.4 on PyPI is a line-by-line transliteration of the JavaScript, and it inherits neither its speed nor its correctness. Measured against 1202 vectors produced by lz-string 1.5.0 on npm — the library the games themselves use:

• 294 vectors compress to different bytes, because it walks Python code points where the format is defined over UTF-16 code units. In practice: an emoji in a player's name is written truncated, and the game reads back U+F600, a private-use box. Every save with an astral character is corrupted on write.
• decompressFromUTF16 raises TypeError on every input. It subtracts an integer from a string. The function has never worked.
• Decompression rebuilds a 65-entry alphabet table per character and reads the bit stream one bit at a time: 22 seconds for a 1.8 MB save, on the request path.

This package is byte-identical to the reference on all 1202 vectors, in all four transports, in both directions, on both backends — and 15–45× faster.

How far that was pushed, because "it round-trips" is not the same as "it is right":

The last row is the one that found bugs — two of them in this package, both invisible to every check above it, and both now pinned by tests. SPEC.md §4 says what they were.

And it is not the only one: py-lzstring 0.1.1, a separate port of the same reference, differs from it on 295 of the same 1202 vectors, first on exactly the same astral character. Two independent Python ports, the same mistake in both.

See SPEC.md for the format itself and for the full divergence table.

Installing

The package is a compiled extension; pip install builds it, and a Rust toolchain is required to do so.

uv add git+ssh://git@github.com/Derfirm/lz-string.git    # or, in a checkout:
uv sync                                                  # builds the extension in place
pip install .                                            # pip works too; maturin does the build

An import without the extension raises rather than falling back to something slower and subtly different.

Build it where it will run. The binary is tied to the C library it was linked against: one built on Debian bookworm (glibc 2.36) will not load on bullseye (2.31), which is what python:3.12.11-slim-bullseye — the image this package is destined for — is built on. It fails at import with GLIBC_2.34 not found, not at build time. Either build inside the target image, or produce a manylinux wheel.

On two real saves, best of three in one run:

Both halves are ours, and the only Rust dependency is pyo3. The obvious alternative, the lz-str crate, was the starting point and did not survive contact with damaged input: it answers the same thing for two failures the reference keeps apart, skips characters the reference reads as zero bits, and refuses a payload whose trailing padding character was trimmed. SPEC.md §7 has the measurements.

A pure-Python implementation of the whole format lives in src/lz_string/_reference.py. It is the test suite's second opinion, not a fallback: nothing imports it at runtime.

Migrating from lzstring

The camelCase API is provided as-is, so the change is one import line:

-from lzstring import LZString
+from lz_string import LZString

Two behaviour changes to know about, both improvements, both in SPEC.md: saves containing astral characters stop being corrupted, and decompressFromUTF16 starts working.

Tests

uv sync --extra dev                      # .python-version pins 3.12, as in production
uv run pytest                            # every test runs against both implementations
uv sync --reinstall-package lzstring-codec   # after touching the Rust: rebuild it
./tools/check_linux.sh                   # built and run inside the target image (needs docker)
./tools/check_linux.sh linux/amd64       # and on production's architecture, cross-compiled

The Linux run is not ceremony. It is done in python:3.12.11-slim-bullseye — the exact image this is destined for — and it has already caught two things a laptop cannot: a test that measured the machine rather than the code, and an extension that built cleanly and then refused to load because it wanted a newer glibc than the image has.

251 tests: the golden corpus per category and transport, the documented behaviour on damaged input, seeded round-trips over random code units and surrogate halves, parity between the shipped extension and the Python reference — including a fuzz of malformed payloads, which is what pins the three corrections the crate underneath needed — and two tests that the extension really does let go of the interpreter while it works.

To regenerate the corpus (deterministic — a diff means the reference moved):

npm --prefix tools install
node tools/gen_vectors.mjs

tools/bench.py reproduces the numbers above.

Releasing

By tag, not by merge. Most merges here are documentation and CI, and PyPI refuses a second upload of the same version — publishing on merge would mean either a version bump per commit or a release job that fails as a matter of course.

# bump `version` in pyproject.toml, note it in CHANGELOG.md, merge, then:
git tag v0.1.0 && git push origin v0.1.0

The workflow builds wheels for linux and macOS on both architectures plus an sdist, checks that the tag and the version in pyproject.toml agree, and only then uploads. workflow_dispatch runs everything except the upload, which is how the pipeline is tested without releasing anything.

The upload needs a credential, and there are two ways to give it one. Either works; the first stores nothing.

Trusted publishing. On PyPI, add a pending publisher for the project name with owner Derfirm, repository lz-string, workflow release.yml, environment pypi. Nothing to configure here — with no PYPI_API_TOKEN secret set, the publish step already exchanges a short-lived OIDC token, which is what it tries today and what fails with "Trusted publishing exchange failure" until the publisher exists.

A token. Create an environment named pypi in the repository settings, put an API token in it as PYPI_API_TOKEN, and re-run the publish job of the tagged release. Protection rules on that environment are where you add an approval step if you want one.

Provenance and licence

MIT (LICENSE). Compression comes from the lz-str crate and the bindings from pyo3, both MIT/Apache-2.0; the decoder is a port of pieroxy/lz-string, MIT, which is also the reference every claim here is measured against. See THIRD_PARTY.md.

No user data lives in this repository: every test input is generated, and the corpus is the reference implementation's own output.