sempress 0.3.1

Semantic compression for tabular data and images using vector quantization

Sempress

Semantic compression for tabular data and images using vector quantization

Sempress achieves 5-15x better compression than gzip on numeric-heavy datasets by learning per-column codebooks with K-Means vector quantization. String and ID columns are preserved losslessly; precision-critical columns can store exact residuals.

Installation

pip install sempress

Optional extras:

pip install sempress[image]   # PSNR/SSIM metrics (scikit-image, scipy)
pip install sempress[audio]   # Audio compression (librosa, soundfile)
pip install sempress[api]     # FastAPI server (fastapi, uvicorn)
pip install sempress[all]     # Everything

CLI Usage

# Compress CSV to .smp format
sempress encode --in data.csv --out data.smp --lock-cols id,timestamp --k 64

# Decompress back to CSV
sempress decode --in data.smp --out restored.csv

# Evaluate reconstruction quality
sempress eval --original data.csv --recon restored.csv --lock-cols id,timestamp

Options:

• --lock-cols: Columns preserved losslessly (strings, IDs, timestamps)
• --residual-cols: Columns with exact error stored (financial, scientific)
• --k: Codebook size per column (default: 64, range: 16-256)
• --uncert-thresh: Flag cells with relative error above threshold (default: 0.2)

Python API

from sempress import encode_csv, decode_to_csv
from sempress.table_encoder import EncodeConfig

config = EncodeConfig(
    lock_cols=["id", "timestamp"],
    residual_cols=["amount"],
    k=64,
    uncertainty_thresh=0.2,
)

# Compress
blob = encode_csv("data.csv", config)
with open("data.smp", "wb") as f:
    f.write(blob)

# Decompress
decode_to_csv(blob, "restored.csv")

How It Works

1. Column analysis - auto-detects numeric vs categorical columns
2. Learn codebooks - K-Means learns k centroids per numeric column
3. Encode to indices - replaces values with nearest centroid index (uint16)
4. Add residuals (optional) - stores exact errors for high-precision columns
5. Package - msgpack + zstd container (.smp / SEMZ1 format) with schema and metadata

Benchmarks

Tested on 10,000 rows of IoT sensor data (1.4 MB):

Metric	Sempress	gzip	Improvement
Compression Ratio	15.72x	2.48x	+533%
Final Size	93 KB	603 KB	84% smaller
Data Fidelity	97.5%	100% (lossless)	Configurable

Sempress excels on numeric-heavy data (IoT, ML features, financial). For text-heavy or very small tables, gzip may be simpler.

Git LFS Integration

For automatic compression in Git repositories, see the companion plugin: git-lfs-sempress

Research Paper

sempress.net/paper.pdf

@article{sempress2025,
  title={Sempress: Semantic Compression for Numeric Tabular Data via Learned Vector Quantization},
  author={Anderson, Keaton},
  year={2025},
  url={https://sempress.net}
}

License

MIT License - see LICENSE for details.