Comparative Study of Different Reduced Precision Techniques in Deep Neural Network

There has been rising interest in reduced precision in the training of deep neural networks (DNNs) that is used from the single-precision (FP32) to different precision format (FP16, FP8, bfloat16) due to the rapid increase in model sizes, which require less representational space when stored in lower precision. However, training a DNN in reduced precision format (FP16, FP8, and bfloat16) is challenging, because the data format may be inadequate for representing the gradients during backpropagation. In this research paper, we compare the various novel approaches to train a DNN using the different reduced precision formats and explore the challenges that we get during the training of DNN in reduced precision. Besides, we also examine various layers in the neural network which make a significant reduction in the backpropagation and also observe that when should get sufficient precision.