poetix-torch 0.3.0

Hybrid Lexicon-Based Streaming Compressor for AI Infrastructure

🚀 Poetix (poetix-torch)

The Engine that Democratized AI Training: Breaking the 2 Million Tokens/Sec Barrier on Consumer Hardware.

Poetix is not just a library; it's a paradigm shift. Designed by Mazen Mohamed (21 years old), this Rust-backed engine was built to solve the ultimate bottleneck in LLM training: Data Starvation.

While others require massive server farms, Poetix allows you to feed your GPUs at professional speeds using nothing but a standard consumer CPU.

---

🏗️ Hardware & Experimental Setup

All benchmarks were executed on a Standard Consumer Machine:

• CPU: Intel Core i5 (Mid-range)
• RAM: Testing included scenarios with only 8GB .
• Model: Experiments performed on high-efficiency training models (~77M Parameters).

The Result? We hit 1.98 Million Tokens/Sec on a single thread. This isn't just code; it's mechanical sympathy between Rust and Hardware.

---

⚡ The Performance Revolution: Rocket vs. Tank

We provide two distinct engines to suit your infrastructure needs:

1. The "Rocket" Mode (In-Memory)

• Speed: 1.98M+ tokens/sec.
• The Trade-off: Loads the entire dataset into RAM.
• Best for: Fast iterations on small-to-medium datasets (up to a few GBs) where speed is the only god.

2. The "Tank" Mode (Infinite Streaming) 🛡️

• Speed: ~770k+ tokens/sec.
• The Magic: ZERO RAM Pressure. Constant memory footprint (< 10 MB).
• Why it's a game-changer: You can train on a 2 Terabyte dataset using a 4GB RAM laptop. It streams data "by the drop" from the disk, ensuring your system never freezes or crashes with Out of Memory (OOM) errors.

---

💻 CLI Command Center

Action	Command	Use Case
Standard Compress	poetix compress data.json data.ptx	Fast, block-based compression.
Infinite Stream	`cat data.json	poetix stream-compress > data.ptx`
Decompress	poetix decompress data.ptx data.json	Restore original data instantly.
Stream Restore	`cat data.ptx	poetix stream-decompress > data.json`

---

🧠 Core Features

• Zero-Copy Architecture: Leveraging Rust's safety and speed to bypass Python's Global Interpreter Lock (GIL).
• Lexicon-Based Hybrid Compression: Combines Aho-Corasick pattern mining with Zstd for up to 90% storage savings.
• PyTorch Integration: Direct injection of torch.Tensor batches into your training loop.

---

📦 Installation

pip install poetix-torch