A distributed training framework for diffusion or consensus-type algorithm.
Project description
Overview
Bluefog is a distributed training framework for Tensorflow and PyTorch based on diffusion/consensus-type algorithm. The goal of Bluefog is to make distributed machine learning fast and fault-tolerant in the heterogeneous environment and users are easy to set up and run experiments without worrying too many low-level details.
REPOSITORY IS STILL A WORK IN PROGRESS.
Philosophy
There are already lots of well-designed and production-level distributed machine learning algorithms, libraries, frameworks, or tools. What is the main different between Bluefog project and others? Why can Bluefog outperform others? Which scenario is more suitable for Bluefog?
Before answering above questions, Demystifying Parallel and Distributed Deep Learning [1] paper has a great conclusion:
The world of deep learning is brimming with concurrency. Even if an aspect is sequential, its consistency requirements can be reduced, due to the robustness of nonlinear optimization, to increase concurrency while still attaining reasonable accuracy, if no better.
The main philosophy is we can sacrifice the consistency or sequential requirement to gain faster trainning speed, more robust system, and more friendly to the heterogeneous enviroment.
[Add more technique details here.]
To comparison with the other algorithms/libraries, we need to demystifying several importance perspectives viewing a distributed machine learning library first.
The Spectrum of Distributed Machine Learning Algorithm
Current machine learning problem is always associated with the large scale of dataset and highly complexity of the model. This provides us lots of ascpects and options to design the algorithm so that the multi-core CPU/GPU in the distributed computation system can be fully exploited. We list the most relevant and important considerations here.
From the aspect of data and modeling concurrency:
Data Parallelism
Model Parallelism
Pipeline Parallelism
The above three techniques are not exclusive to each other. For example, tensorflow allow users can utilize all three techniques at the same time. The Bluefog project focused on Data Parallelism. One reason, of course, is the base algorithm derived based on the assumption that the dataset is distributed over different nodes. But, more importantly, among these three techniques, data parallelism is the most popular approaches thanks to its excellent scalability and flexibility on almost any model.
From the aspect of communication architecture:
Parameter Server(PS) —- (Distributed but still centralized)
Sharded PS
Hierarchical PS
Peer-to-Peer —– (Distributed but also decentralized)
Ring-AllReduce
Neighbor-Collective
Apparently, Bluefog project belongs to the peer-to-peer model. Multiple nodes/machines will distributedly and no centralized node will gather all the informations.
From the aspect of parameter consistency:
In the distributed learning system, parameter consistency means the similarity between the parameter stored in the local machine. We list five typical algorithms from strongest consistency to weakest consistency.
Model Replication
Delayed Updating (like asynchronous algorithm.)
Model Averaging
Ensemble Learning
Bluefog project focused on the asynchronous training through the diffusion/consensus algorithm, which is one kind of model averaging algorithms. The parameter learned in different nodes through the Bluefog algorithm are slightly different. But unlike ensemble learning, all nodes are highly similar.
From the aspect of updating synchronization:
Synchronous updating
Stale-Synchronous updating
Asynchronous updating
Typically, the more “asynchronous” updating, the faster on the training. However, we will loss the parameter consistency.
From the aspect of information fusion:
Averaging over the gradients
Averaging over the parameter/iterates
Averaging over the dual variable
Most strong consistency algorithm is averaging over the gradients. However, Bluefog project is averaging over the parameter directly. One advantage of averaging over the parameter is the resilient on the noise or error. Also, noticing these three methods are not exclusive.
From the aspect of reducing communication cost:
Temporal compression (Fine- vs Coarse-Grained Fusion)
Spatial compression (Sparse/Sliced tensors)
Btye compression (Quantization)
Neighbor compression (Selecting less neighbors)
We don’t have any implementation to support it yet. We do plan to support it in the future.