Minimal Dependency Dataset Library for PyTorch

Bachelor Project — DFKI (German Research Center for Artificial Intelligence) | SS 2026

Project Description

Training large deep learning models on GPU clusters requires efficient data pipelines. When datasets reside on remote network storage, loading and preprocessing can become the primary bottleneck, leaving expensive GPUs underutilized. Existing solutions (WebDataset, MosaicML StreamingDataset, TFRecord) often require extensive dependencies and major changes to training code.

The goal of this project is to analyse performance bottlenecks in the data loading pipeline, then design and implement a simple library for storing and loading training data from a remote storage server. The library should be minimal — relying only on PyTorch and the Python standard library — while supporting multi-threaded loading, lock-free sampling, distributed training (DDP), and built-in instrumentation for performance profiling.

Project Goals & Status

Key Results

Storage Format Evaluation (100k images, tiny-imagenet, fscratch SSD)

DataLoader Performance (Parquet, A100 CPU node, 100k images)

GPU Compute Delays (ResNet-50, batch_size=256, SGD)

Repository Structure

bachelor-project/
├── minimal_dataset/ # Library module
│ ├── init.py # Public API
│ ├── dataset.py # BaseDataset (ImageFolder reader)
│ ├── parquet_dataset.py # ParquetDataset (Parquet reader)
│ ├── sampler.py # LockFreeSampler (lock-free index partitioning)
│ ├── dataloader.py # DataLoader (multi-threaded, staging + batch queues)
│ ├── monitored_queue.py # MonitoredQueue (queue instrumentation)
│ └── metrics.py # MetricsTracker + WorkerMetrics
├── benchmarks/
│ ├── gpu/ # GPU benchmarking & dummy model
│ │ ├── benchmark_gpu.py
│ │ ├── calibrate_dummy.py
│ │ ├── dummy_model.py
│ │ ├── build_delay_config.py
│ │ └── gpu_delays.json
│ └── storage/ # Storage format evaluation
│ ├── benchmark_plain.py
│ ├── benchmark_zip.py
│ ├── benchmark_tar.py
│ ├── benchmark_lmdb.py
│ ├── benchmark_parquet.py
│ ├── benchmark_msgpack.py
│ └── file_io.py
├── tests/ # DataLoader tests & benchmarks
│ ├── test_dataloader.py
│ ├── benchmark_dataloader.py
│ ├── benchmark_pytorch.py
│ ├── plot_comparison.py
│ └── launch_full_benchmark.sh
├── training/ # ResNet training scripts
│ ├── train_resnet50.py
│ ├── train_resnet50_ddp.py
│ └── analyze_dataloading.py
├── docs/
│ ├── images/ # Architecture diagrams & benchmark plots
│ │ ├── 00_MainOverview1.png
│ │ ├── 02_dataLoader1.png
│ │ ├── 04_Storage Format Evaluation.png
│ │ ├── 05_GPU Compute Delay.png
│ │ ├── plot_ours_throughput.png
│ │ ├── plot_pytorch_throughput.png
│ │ ├── plot_comparison_bs16.png
│ │ └── plot_speedup_comparison.png
│ └── architecture.md
└── README.md