dataloader-param-helper 0.1.1

Finding the optimal parameters for a Dataloader.

⚡ DataLoader Parameter Optimization Package

🚀 Optimize torch.utils.data.DataLoader Parameters Effortlessly

Finding the optimal configuration for torch.utils.data.DataLoader parameters such as num_workers, prefetch_factor, pin_memory, and persistent_workers can be challenging and system-dependent. This package leverages efficient algorithms to streamline the process, saving you time and effort while enhancing your model's training speed.

Overview

This package is designed to optimize the loading parameters for torch.utils.data.DataLoader, specifically num_workers, prefetch_factor, pin_memory, and persistent_workers.

These parameters can have varying optimal values depending on the specific hardware and system configurations.

Manually testing all possible combinations to determine the fastest configuration is often an extremely time-consuming and labor-intensive process. This package simplifies the task by employing advanced techniques such as binary search, early termination, and time prediction algorithms to identify the most efficient parameters with minimal testing.

While the identified configuration may not always guarantee absolute optimality, it is designed to outperform default settings and significantly accelerate your training pipeline, ensuring your model runs more efficiently.

Caching and Loading Optimized Parameters

The process of identifying the optimal parameters may take some time to complete, as it involves thorough testing and analysis. However, this package includes a built-in caching and loading mechanism that automatically saves the results after the first test run. In subsequent runs, the saved parameters are loaded automatically, significantly reducing the time required for repeated executions.

If desired, this caching feature can be disabled for manual parameter testing. Additionally, after running the tests, you can inspect the saved results file to review the optimized parameters and their corresponding configurations.

Installation

You can install this package directly from PyPI using pip. Follow the command below to install:

pip install dataloader-param-helper