KolmoX

Architectural Positioning: KolmoX is not a general-purpose replacement for Zstandard, Snappy, or Deflate. Exactly as the PNG Delta Filter or the FLAC Mid/Side Decorrelator operate ahead of Deflate, KolmoX is a domain-specific structural preconditioner. It linearizes physical coordinates, IEEE-754 mantissas, and multi-dimensional matrices before entropy coding, eliminating topological correlations that sliding-window (LZ77) engines cannot detect.

For production workloads, use the dedicated engine classes directly (FitsEngine, ScientificFloatEngine, GCodeEngine, etc.). An Adaptive Competitive Fallback ensures KolmoX output is mathematically guaranteed never to exceed plain Zstandard.

Evidence Tiers

• ✅ Tier 1 (Reproducible In-Script): Synthetic-but-realistic datasets generated programmatically via python benchmarks/benchmark_extended.py. Fully reproducible by any third party with zero external dependencies.
• 📋 Tier 2 (Production Data / Reported): Benchmarked against production binary streams (NASA/STScI James Webb MIRI FITS, 5-axis CNC toolpaths, and Modbus/PLC telemetry registers). Reproducible via python benchmarks/download_datasets.py for publicly licensed archives.

Why KolmoX Matters: Today and in the Multi-Petabyte Future

We live in an era where data generation has outpaced network bandwidth and storage interconnect speeds.

• The Scientific & AI Bottleneck: Modern AI pipelines, LLM checkpointing, physics engines, and space telescopes (JWST, Roman) generate billions of IEEE-754 floating-point numbers. Standard compressors choke on shot noise and mantissa entropy. KolmoX delivers up to 15.7x on binary registers and 1.97x on dense vectors and +22.8% net savings over Zstd on raw JWST FITS datasets without altering a single bit.
• Smart Manufacturing & Industry 4.0: Robotics, CNC machining, 3D additive manufacturing, and autonomous vehicle LiDAR streams churn out terabytes of continuous telemetry daily. Squeezing columnar telemetry by 12x–16x drastically cuts cloud egress bills and edge-to-cloud transmission latency.
• Lossless is Non-Negotiable: In medical imaging, astrophysics, engineering CAD, industrial telemetry, and legal compliance, lossy compression artifacts are unacceptable. KolmoX proves that "lossless" doesn't have to mean "poor compression ratios".

Real-World Benchmark Results

All tests certify exact mathematical data restoration (zero precision loss, bit-exact roundtrip):

Methodology: every row above was measured on 2026-09-03 by running the data through the public KolmoXPipeline.compress_bytes() / decompress_bytes() API - the same path a user gets - with a bit-exact roundtrip assertion on each. Figures produced by calling engine classes directly, outside the pipeline, are not reported here: they omit the 24-byte KMX2 container header and bypass the adaptive competitive fallback. Every † row is reproducible with python benchmarks/benchmark_extended.py, which regenerates each dataset from a fixed seed and re-asserts every roundtrip. Compression ratios are deterministic; the throughput columns are single-run samples and vary by roughly ±15% between runs.

† Synthetic dataset, generated programmatically. Only the Astrophysics FITS row uses a real-world production capture (a 117 MB NASA/STScI JWST MIRI observation of the Carina Nebula, compressed whole-file through the automatic domain router). Extracting just the pixel plane with astropy and compressing that instead yields 1.26x → 1.61x (+21.41%).

¹ The LiDAR dataset is a deterministic arithmetic ramp, which is why even the plain Zstd baseline reaches 27.11x. It demonstrates the transform works, but it is not representative of real scanner output - treat this row as a mechanism check, not a performance claim.

² A 40 KB synthetic instruction stream; at that size the throughput timings are dominated by measurement noise.

Domain gains depend on the data actually having the structure the transform exploits. On a high-entropy input - a random-walk mesh with 6 decimals of noise per coordinate, or the x86 stream above - the transform yields nothing, the adaptive competitive fallback discards it, and the stored result is the plain Zstd baseline plus the 24-byte KMX2 header. The raster decompression figure (~2.4 MB/s) reflects a pure-Python pixel loop whose sequential dependency has not yet been ported to the C extension; it is the current honest number, not a target.

Installation & Setup

# Clone the repository
git clone https://github.com/Toskurim/KolmoX.git
cd KolmoX

# Install Python package in editable mode (with C-extension)
pip install -r requirements.txt
pip install -e .

CMake Standalone C-Library Build

For direct C/C++/Rust integration without Python:

cmake -B build
cmake --build build --config Release

This generates libkolmox.so (or kolmox.dll / libkolmox.dylib) using the public C-ABI defined in include/kolmox.h.

Quickstart CLI

# Auto-detect domain and compress into KMX2 container
kolmox compress path/to/file.gcode -o file.kmx

# Bit-exact decompression
kolmox decompress file.kmx -o restored.gcode

# Run throughput benchmark suite
python tests/benchmark_throughput.py

Constant-Memory Streaming API

For multi-Gigabyte files, KolmoX provides a streaming engine that executes in bounded RAM (default 8MB chunks):

from kolmox.core.streaming import KolmoXStreamer

streamer = KolmoXStreamer(chunk_size=8 * 1024 * 1024)

# Stream compression from file-like objects
with open("large_scan.xyz", "rb") as src, open("large_scan.kmxs", "wb") as dst:
    streamer.compress_stream(src, dst, filename="large_scan.xyz")

Testing & Fuzz Resilience

KolmoX includes a 33-test validation suite covering end-to-end roundtrips, C-ABI equivalence, astrophysics FITS preconditioning, and fuzz resilience against malformed or truncated data:

pytest tests/ -v

Technical Whitepaper

A full academic whitepaper detailing the Kolmogorov complexity foundations, KMX2 container specification, and mathematical transforms is available in docs/WHITEPAPER.md and as a downloadable PDF in docs/KolmoX_Technical_Paper_v1.1.0_Complete_EN.pdf.

License

KolmoX is dual-licensed:

• Open Source: GNU Affero General Public License v3.0 (AGPLv3).
• Commercial & Enterprise: For proprietary software integration without AGPLv3 copyleft obligations, contact toskurim@gmail.com.