ntcompress

Pure-Python, standard-library-only implementations of every compression format used by Microsoft's Extensible Storage Engine (ESE/ESENT, the "JET Blue" engine behind NTDS.dit, Exchange stores, Windows.edb, SRUDB.dat, and WebCacheV01.dat) and Windows ntdll.dll compression (RtlCompressBuffer / RtlDecompressBuffer). No runtime dependencies beyond the standard library.

Scope

This library covers two distinct compression layers used by Windows:

ESE record compression (vs esent.dll)

ESE stores a one-byte header on each compressed record cell; the top 5 bits select the scheme (first_byte >> 3) and the low 3 bits carry scheme-specific flags.

*XPRESS9 encoder matches the MIT ESE C reference byte-for-byte excluding the non-deterministic session signature (CRC32(__rdtsc())), which even esent.dll varies on every call. **XPRESS10 verified byte-identical by component construction (LZ4 block + CRC-32C + CRC-64/NVME each individually proven); not triggerable on commodity hardware (requires Corsica/QAT flight flag).

ntdll compression (vs RtlCompressBuffer)

Windows ntdll.dll exposes a separate set of CompressionFormatAndEngine IDs used by SMB, NTFS, RDP, and other protocols. These use raw bitstreams with no ESE framing.

Format 0x0005 is an undocumented XPRESS9-family Huffman LZ77 codec (magic 0xC039E510) with a compact 10-byte header, reverse-engineered from ntdll.dll Build 20348. It uses the same core algorithm as ESE XPRESS9 (magic 0x4E86D72A) but with different outer framing. Format 0x0006 (XP10) is the same LZ4 block codec used by ESE scheme 0x6 and 0x7, without the ESE header.

Design principles

• Standard library only at runtime. No third-party dependencies. zlib/lz4 are deliberately not used; the CRCs and the LZ4 block codec are implemented directly.
• Two subpackages, one interface pattern. ntcompress.ese for ESE record-compression, ntcompress.ntdll for ntdll standalone codecs. Both offer Shape A (enum dispatch) and Shape B (direct module import) APIs.
• Traceable to a spec or a source line. Every wire constant cites [MS-XCA] §x.y or an ESE file:line. Where the shipped C diverges from the spec text, both are documented.
• Byte-identical verification. Compress output is verified against gold vectors captured from esent.dll and ntdll.dll on 16 Windows builds (XP SP3 through Server 2025).
• Typed and documented. Python 3.11+, full type hints, Google-style docstrings, ruff-clean under select = ["ALL"], ty-clean.

Installation

Install from PyPI:

uv add ntcompress        # recommended
pip install ntcompress    # or with pip

Or install from source:

uv add git+https://github.com/StrongWind1/ntcompress

Requires Python >= 3.11.

Usage

ESE record compression

import ntcompress.ese as ese

# Decompress a cell (format is in byte 0)
plaintext = ese.decompress(cell)
size = ese.decompressed_size(cell)

# Compress with a specific format
cell = ese.compress(plaintext, ese.Format.XPRESS)
cell = ese.compress(plaintext, ese.Format.LZ4)
cell = ese.compress(plaintext, ese.Format.XPRESS9)

# Or use Shape B (direct module access)
from ntcompress.ese import xpress
cell = xpress.compress(plaintext)
plaintext = xpress.decompress(cell)

ntdll standalone codecs

import ntcompress.ntdll as ntdll

# Shape A (enum dispatch)
compressed = ntdll.compress(data, ntdll.Format.LZNT1)
plaintext = ntdll.decompress(compressed, ntdll.Format.LZNT1)

# Windows constant aliases
compressed = ntdll.compress(data, ntdll.COMPRESSION_FORMAT_XPRESS_HUFF)

# Shape B (direct module access)
from ntcompress.ntdll import lznt1, xpress_huff, xpress9, xp10, deflate
from ntcompress.ntdll import zlib as ntdll_zlib

plaintext = lznt1.decompress(compressed_stream)
compressed = xpress_huff.compress(data)
plaintext = xpress9.decompress(stream)         # ntdll 0x0005 (compact XPRESS9)
compressed = xp10.compress(data)               # ntdll 0x0006 (LZ4 block)
compressed = deflate.compress(data)            # ntdll 0x0007
compressed = ntdll_zlib.compress(data)         # ntdll 0x0008

License and attribution

Apache-2.0 (LICENSE). The XPRESS9 codec is an attributed port of Microsoft's MIT-licensed ESE source; the Plain LZ77, Xpress Huffman, and LZNT1 codecs are derived from the published [MS-XCA] specification; the LZ4 block codec follows the public LZ4 format. Full notices are in THIRD-PARTY-NOTICES.md.