cdmltrain 1.0.0

Stream ML datasets from ZIP/ZSTD archives into PyTorch without disk extraction.

cdmltrain 🚀

Stream ML Datasets Directly from ZIP / ZSTD Archives — No Extraction. No Wasted Storage. Zero OOM.

---

🔥 The Problem This Solves

Every Data Scientist / ML Engineer hits this wall:

"Your 100 GB Kaggle dataset is a ZIP file.
 Extracting it takes 2 hours and needs 300 GB of free disk space.
 Your Colab/Kaggle notebook crashes with Out-of-Memory errors."

cdmltrain eliminates this problem entirely.

It lets PyTorch read images, audio, text, CSV, JSON — any data — directly from a compressed archive into RAM, skipping disk extraction completely.

---

✨ Key Features

Feature	Description
🗜️ Format Agnostic	Images, audio (.wav), text, CSV, JSON, binary — all supported
⚡ O(1) Random Access	ZIP Central Directory indexing — jumps to any file instantly
🧠 Memory-Safe Cache	Custom LRU cache enforces strict RAM limits — zero OOM crashes
🔒 Thread Safe	Concurrent reads for PyTorch DataLoader(num_workers=N)
🔧 3-Tier Architecture	Auto-selects best engine based on your hardware
🏎️ ZSTD Support	.tar.zst archives — 25x faster than ZIP deflate
🎮 GPU Direct Loader	Streams data to CUDA VRAM with async prefetch
🌍 Cross-Platform	Windows, Linux, macOS — works everywhere

---

🏗️ Architecture (3 Tiers — Auto-Selected)

Your ZIP/ZSTD File
        │
        ▼
┌─────────────────────────────────────────────────────┐
│  Tier 1: Python CoreStreamEngine                    │
│  ✅ Works on ANY machine, no dependencies            │
│  → O(1) Index + LRU Cache + Thread Safety           │
├─────────────────────────────────────────────────────┤
│  Tier 2: C++ FastCoreEngine (pybind11)              │
│  ✅ Auto-enabled if C++ Build Tools installed        │
│  → Bypasses Python GIL, faster multi-worker reads   │
├─────────────────────────────────────────────────────┤
│  Tier 3: ZSTD Engine + GPU Direct Loader            │
│  ✅ pip install zstandard   (for .tar.zst files)     │
│  ✅ NVIDIA GPU (for direct VRAM streaming)           │
│  → 25x faster decompression + near-zero GPU latency │
└─────────────────────────────────────────────────────┘
        │
        ▼
  PyTorch DataLoader → Model Training

The library auto-detects your hardware and picks the best tier. You write the same code regardless.

---

📦 Installation

Note: PyPI release coming soon. Install locally for now.

# Step 1: Clone the repo
git clone https://github.com/prem85642/cdmltrain.git
cd cdmltrain

# Step 2: Install
pip install .

# Step 3 (Optional): ZSTD support for .tar.zst archives
pip install zstandard

C++ Acceleration (Tier 2):

• Windows: Install Microsoft C++ Build Tools
• Linux: sudo apt-get install build-essential
• If skipped: pure-Python engine runs automatically — no errors.

---

🚀 Quick Start

Basic Usage (ZIP — Any Data Type)

from cdmltrain import CDMLStreamDataset
from torch.utils.data import DataLoader

# Point directly to your ZIP — no extraction needed!
dataset = CDMLStreamDataset(
    zip_path="my_dataset.zip",
    max_cache_mb=2048    # RAM cache limit (MB)
)

dataloader = DataLoader(dataset, batch_size=32, shuffle=True, num_workers=4)

for batch in dataloader:
    # Train your model normally
    pass

Image Dataset (with PyTorch Transforms)

from cdmltrain import CDMLStreamDataset
from torchvision import transforms

transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize([0.5], [0.5])
])

dataset = CDMLStreamDataset("images.zip", transform=transform, is_image=True)

ZSTD Archive (25x Faster Decompression) ⚡

# Just change the file extension — everything else is identical!
dataset = CDMLStreamDataset("my_dataset.tar.zst", max_cache_mb=2048)

GPU Direct Loader (NVIDIA VRAM Streaming) 🎮

For the highest possible performance, cdmltrain bypasses PyTorch's DataLoader multiprocessing bottleneck using CUDA Pinned Memory:

from cdmltrain.gpu_loader import GPUDirectLoader

# Stream directly into GPU:0 VRAM using asynchronous DMA prefetching
loader = GPUDirectLoader(dataset, batch_size=64, device="cuda:0")

for batch in loader:
    # `batch` is already a CUDA Tensor! No need for `.to(device)`
    loss = model(batch) 

---

🔬 Data Scientist Debugging Tools

When a deep learning model crashes on Epoch 4, Batch 1400 because of a corrupt image inside the archive, you don't need to extract the 100GB ZIP to find it. cdmltrain provides instant debugging hooks:

1. Extract a Corrupt File by Index

Instantly extract the broken image to your disk to inspect it visually:

# Instantly fetch image #1400 and save it to the current folder
dataset.extract_sample_to_disk(idx=1400, export_path="./debug_folder/")
# Output: [CDML] Extracted file 'broken_dog.jpg' to -> ./debug_folder/broken_dog.jpg

2. Fetch a specific file by its Name

Want to look at the labels.csv file without touching the millions of images inside the ZIP?

# Returns raw bytes (or a PIL Image if is_image=True)
csv_bytes = dataset.get_by_filename("train_labels.csv")
print(csv_bytes.decode('utf-8')[:100])

---

🖥️ 3-Tier Architecture Fallback

cdmltrain auto-detects your system capabilities and selects the fastest engine:

⚙️ Configuration Parameters

Parameter	Type	Default	Description
zip_path	str	(required)	Path to .zip or .tar.zst file
transform	callable	None	PyTorch/torchvision transform
is_image	bool	False	True enables PIL image decoding
max_cache_mb	int	2048	Max RAM for caching (MB)

max_cache_mb Guide:

Your RAM	Recommended
4 GB (Colab Free)	512
8 GB (Laptop)	2048
16 GB (PC)	6000
32 GB+ (Server)	16000

---

📊 Benchmarks (Real Tests)

ZSTD vs ZIP Speed (200 files × 10KB)

Engine	Speed	Speedup
ZIP (Deflate) — Tier 1/2	29,204 files/sec	baseline
ZSTD — Tier 3	739,653 files/sec	🔥 25x faster

GPU Direct Loader (Google Colab T4)

Metric	Value
GPU	Tesla T4 (15.6 GB VRAM)
Batch device	cuda:0 — data streamed to VRAM
Throughput	17,512 items/sec
Epoch time (100 items)	0.0045s

Memory Safety Test

Test	Result
Cache limit: 1MB, Data: 2MB	✅ Stayed under 1MB
Thread safety: 8 workers	✅ Zero race conditions
Corrupted ZIP	✅ Rejected cleanly
50MB single file	✅ Byte-exact in 0.108s
1000-file archive	✅ Indexed in 0.04s

---

🐛 Debugging / Inspection

Inspect any specific file without unzipping:

dataset.extract_sample_to_disk(idx=42, export_path="./inspection/")

---

🛠️ Troubleshooting

ModuleNotFoundError: No module named 'cdmltrain'

git clone https://github.com/prem85642/cdmltrain.git && cd cdmltrain && pip install .

ModuleNotFoundError: No module named 'zstandard'

pip install zstandard

Microsoft Visual C++ 14.0 required (Windows)

Install C++ Build Tools. Or skip — pure Python engine works fine.

Out of Memory on Colab

dataset = CDMLStreamDataset("data.zip", max_cache_mb=512)  # Reduce cache

Bad CRC-32 error

python -c "import zipfile; print(zipfile.ZipFile('file.zip').testzip())"
# None = healthy, anything else = re-download

PIL.UnidentifiedImageError

dataset = CDMLStreamDataset("data.zip", is_image=False)  # Not an image dataset

---

🖥️ OS Compatibility

Feature	Windows	Linux	macOS
ZIP Engine (Tier 1)	✅	✅	✅
ZSTD Engine (Tier 3)	✅	✅	✅
GPU Direct Loader	✅	✅	✅
C++ Fast Engine (Tier 2)	✅ pre-built	✅ compile via pip install .	✅ compile via pip install .

---

📁 Project Structure

cdmltrain/
├── cdmltrain/
│   ├── __init__.py          # Package entry point
│   ├── core.py              # Tier 1: Python CoreStreamEngine
│   ├── dataset.py           # CDMLStreamDataset (auto-tier selection)
│   ├── zstd_engine.py       # Tier 3: ZSTD streaming engine
│   ├── gpu_loader.py        # Tier 3: GPU Direct Loader (CUDA pinned memory)
│   └── src/
│       └── fast_core.cpp    # Tier 2: C++ FastCoreEngine (pybind11)
├── demo.py                  # Quickstart demo
├── quickstart.ipynb         # Jupyter Notebook tutorial
├── test_enterprise_audit.py # Enterprise QA suite (8 tests)
├── test_zstd_benchmark.py   # ZSTD vs ZIP benchmark
├── test_zstd_compat.py      # Cross-format compatibility test
├── test_gpu_loader.py       # GPU Direct Loader test
├── setup.py                 # pip install configuration
├── requirements.txt         # Dependencies
└── LICENSE                  # MIT License

---

🤝 Contributing

Pull requests are welcome! For major changes, please open an issue first.

---

📄 License

MIT License — see LICENSE for details.

Made with ❤️ for the ML community — because your model matters more than your storage bill.