Tracking, debugging, and patching non-determinism in TensorFlow
This repository serves three purposes:
- Provide up-to-date information (in this file) about non-determinism sources and solutions in TensorFlow and beyond, with a focus on determinism when running on GPUs.
- Provide a patch to attain various levels of GPU-specific determinism in
stock TensorFlow, via the installation of the
- Be the location where a TensorFlow determinism debug tool will be released
as part of the
For more information, please watch the video of the GTC 2019 talk Determinism in Deep Learning. The desciption under that video also includes links to the slides from the talk and to a poster presentation on this topic.
pip to install:
pip install tensorflow-determinism
This will install a package that can be imported as
tensorflow-determinism will not automatically install
TensorFlow. The intention of this is to allow you to install your chosen
version of TensorFlow. You will need to install your chosen version of
TensorFlow before you can import and use
Deterministic TensorFlow Solutions
There are currently two main ways to access GPU-deterministic functionality in TensorFlow for most deep learning applications. The first way is to use an NVIDIA NGC TensorFlow container. The second way is to use version 1.14, 1.15, or 2.0 of stock TensorFlow with GPU support, plus the application of a patch supplied in this repo.
The longer-term intention and plan is to upstream all solutions into stock TensorFlow.
Determinism is not guaranteed when XLA JIT compilation is enabled.
NVIDIA NGC TensorFlow Containers
NGC TensorFlow containers, starting with version 19.06, implement GPU-deterministic TensorFlow functionality. In Python code running inside the container, this can be enabled as follows:
import tensorflow as tf import os os.environ['TF_DETERMINISTIC_OPS'] = '1' # Now build your graph and train it
The following table shows which version of TensorFlow each NGC container version is based on:
|NGC Container Version||TensorFlow Version|
|19.07 - 19.09||1.14|
For information about pulling and running the NVIDIA NGC containers, see these instructions.
Versions 1.14, 1.15, and 2.0 of stock TensorFlow implement a reduced form of GPU
determinism, which must be supplemented with a patch provided in this repo.
The following Python code is running on a machine in which
tensorflow-gpu=2.0.0 has been installed correctly and on which
tensorflow-determinism has also been installed (as shown in the
installation section above).
import tensorflow as tf from tfdeterminism import patch patch() # use tf as normal
Stock TensorFlow with GPU support can be installed as follows:
pip install tensorflow-gpu=2.0.0
The TensorFlow project includes detailed instructions for installing TensorFlow with GPU support.
Additional Ingredients in the Determinism Recipe
You'll also need to set any and all appropriate random seeds:
os.environ['PYTHONHASHSEED']=str(SEED) random.seed(SEED) np.random.seed(SEED) tf.set_random_seed(SEED)
If you're using Horovod for multi-GPU training, you may need to disable Tensor Fusion (assuming that the non-determinism associated with Tensor Fusion has not yet been resolved):
Detailed Status of Determinism in TensorFlow and Beyond
Confirmed and likely sources of non-determinism, along with any existing solutions, are being tracked here.
GPU-Specific Sources of Non-Determinism
Historic GPU-Specific Sources of Non-Determinism
In the past,
non-deterministically when running on a GPU. This was resolved before
TensorFlow version 1.12. These ops now function deterministically
by default when running on a GPU.
Confirmed Current GPU-Specific Sources of Non-Determinism (With Solutions)
|Source||NGC 19.06+ / TF 2.1||TF 1.14, 1.15, 2.0|
|TF auto-tuning of cuDNN convolution algorithms||TCD or TDO||TCD or TDP|
|cuDNN convolution backprop to weight gradients||TCD or TDO||TCD or TDP|
|cuDNN convolution backprop to data gradients||TCD or TDO||TCD or TDP|
|cuDNN max-pooling backprop||TCD or TDO||TCD or TDP|
Key to the solutions refenced above:
|TCD||Set environment variable
|TDO||Set environment variable
|NS1||There is currently no solution available for this, but one is under development.|
- XLA: These solutions will not work when XLA JIT compilation is enabled.
Other Possible GPU-Specific Sources of Non-Determinism
Going beyond the above-mentioned sources, in version 1.12 of TensorFlow (and
also in the master branch on 2019-03-03, afer release 1.31.1), the following
files call CUDA
atomicAdd either directly or indirectly. This makes them
candidates for the injection of non-determinism.
Unless you are using TensorFlow ops that depend on these files (i.e. ops with similar names), then your model will not be affected by these potential sources of non-determinism.
atomicAdd, there are ten other CUDA atomic functions whose use
could lead to the injection of non-determinism, such as
atomicCAS (the most
generic, atomic compare and swap). Note also that the word 'atomic' was present
in 167 files in the TensorFlow repo and some of these may be related to the use
of CUDA atomic operations. It's important to remember that it's possible to use
CUDA atomic operations without injecting non-determinism, and that, therefore,
when CUDA atomic operations are present in op code, it doesn't guarantee that
the op injects non-determinism into the computation.
Sources of Non-Determinism in TensorFlow Unrelated to GPU
- Issue 29101: Random
seed not set in graph context of
Dataset#map. This may have been resolved in version 1.14 of TensorFlow.
tf.data.Datasetwith more than one worker. The work-around is to use only one worker.
Sources of Non-Determinism Beyond TensorFlow
- TensorRT timing-based kernel schedule. Each time an inference engine is generated, it could be slightly different, particularly if there is varying load on the machine used to run TensorRT. There is a solution planned for this.
- Horovod Tensor Fusion. Work-around: disable Tensor Fusion by setting the
HOROVOD_FUSION_THRESHOLDto '0'. This issue may have been resolved by Horovod pull-request 1130 (not yet confirmed).
This section catalogs relevant links.
|2652||Backward pass of broadcasting on GPU is non-deterministic||2019-10-08|
|2732||Mention that GPU reductions are nondeterministic in docs||2019-10-08|
|16889||Problems Getting TensorFlow to behave Deterministically||2019-10-08|
|18096||Feature Request: Support for configuring deterministic options of cuDNN conv routines||2019-10-08|
|29101||Random seed not set in graph context of
TensorFlow Pull Requests
|10636||Non-determinism Docs||closed (not merged)||2019-10-08|
|24273||Enable dataset.map to respect seeds from the outer context||closed (not merged)||N/A|
|24747||Add cuDNN deterministic env variable (only for convolution).||merged pre-1.14||N/A|
|25269||Add deterministic cuDNN max-pooling||merged pre-1.14||N/A|
|25796||Added tests for
|29667||Add release note about
||merged into r1.14||N/A|
|31389||Enhance release notes related to
||merged into r1.14||N/A|
|32979||Fix typo in release note||closed (not merged)||N/A|
|33483||Fix small typo in v2.0.0 release note||N/A|
- Two Sigma: A Workaround for Non-Determinism in TensorFlow
- Keras issue 12800: Unable to get reproducible results using Keras with TF backend on GPU (updated on 2019-10-08)
- PyTorch Reproducibility (from the official documentation)
- Chainer PR 2710: cuDNN Deterministic mode
- Stack Overflow: Tensorflow: Different results with the same random seed
- Stack Overflow: Are tensorflow random values guaranteed to be the same inside a single run? (comment) (updated 2019-10-10).
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size tensorflow-determinism-0.3.0.tar.gz (12.3 kB)||File type Source||Python version None||Upload date||Hashes View hashes|
Hashes for tensorflow-determinism-0.3.0.tar.gz