wrdz

Dictionary-based compression for short text strings and URLs.

Key Features

• Optimized for Short Text: Designed for strings under 140 characters
• URL Compression: Special dictionary trained on URL patterns
• UTF-8 Support: Handles any valid UTF-8 text
• Domain Adaptable: Train custom dictionaries for your use case
• Pure Python: No external dependencies for runtime

Quick Start

pip install wrdz

from wrdz import compress, decompress

# English text compression
text = "The quick brown fox jumps over the lazy dog"
compressed = compress(text)
print(f"Original: {len(text)} bytes")
print(f"Compressed: {len(compressed)} bytes")
print(f"Ratio: {len(compressed)/len(text):.2%}")

# URL compression
from wrdz import compress_urls, decompress_urls

url = "https://github.com/pjwerneck/wrdz"
compressed = compress_urls(url)
print(f"Original: {len(url)} bytes")
print(f"Compressed: {len(compressed)} bytes")
print(f"Ratio: {len(compressed)/len(url):.2%}")

Training Custom Dictionaries

from wrdz.train import train_dictionary, save_dictionary

# Train on domain-specific text
cbook, dbook = train_dictionary(
    text="your training data",
    max_sub_len=4,    # Max sequence length
    dict_size=8192    # Dictionary entries
)

# Save for reuse
save_dictionary(cbook, dbook, "domain.dict")

# Use in compression
from wrdz.base import base_compress, base_decompress

compressed = base_compress("text", cbook)
original = base_decompress(compressed, dbook)

Compression Benchmarks

The tables below show compression ratios for different dictionary sizes and maximum sequence lengths. The Δ% column shows the improvement over the baseline compression ratio.

US English

The en_US dictionary is trained on a 1M lines subset of the cnn_dailymail dataset.

URLs

The urls dictionary is trained on the ada-url dataset.

Technical Details

1. Dictionary Training

   • Analyzes frequency of character sequences in training data
   • Selects sequences that maximize compression
   • Assigns variable-length binary codes based on frequency

2. Compression Format

   • 1-bit flag per token (dictionary/raw)
   • Variable-length dictionary codes (4-14 bits)
   • Raw UTF-8 encoding with 2-bit length prefix

License

MIT License. See LICENSE file for details.