Back to Search
Start Over
Mixed Precision Block Fused Multiply-Add: Error Analysis and Application to GPU Tensor Cores
- Source :
- SIAM Journal on Scientific Computing, SIAM Journal on Scientific Computing, Society for Industrial and Applied Mathematics, 2020, 42 (3), pp.C124-C141. ⟨10.1137/19M1289546⟩
- Publication Year :
- 2020
- Publisher :
- HAL CCSD, 2020.
-
Abstract
- International audience; Computing units that carry out a fused multiply-add (FMA) operation with matrix arguments, referred to as tensor units by some vendors, have great potential for use in scientific computing. However, these units are inherently mixed precision and existing rounding error analyses do not support them. We consider a mixed precision block FMA that generalizes both the usual scalar FMA and existing tensor units. We describe how to exploit such a block FMA in the numerical linear algebra kernels of matrix multiplication and LU factorization and give detailed rounding error analyses of both kernels. An important application is to GMRES-based iterative refinement with block FMAs, for which our analysis provides new insight. Our framework is applicable to the tensor core units in the NVIDIA Volta and Turing GPUs. For these we compare matrix multiplication and LU factorization with TC16 and TC32 forms of FMA, which differ in the precision used for the output of the tensor cores. Our experiments on an NVDIA V100 GPU confirm the predictions of the analysis that the TC32 variant is much more accurate than the TC16 one, and they show that the accuracy boost is obtained with almost no performance loss.
- Subjects :
- fused multiply-add
Floating point
floating-point arithmetic
Carry (arithmetic)
matrix multiplication
010103 numerical & computational mathematics
01 natural sciences
law.invention
Computational science
Matrix (mathematics)
law
Tensor (intrinsic definition)
[INFO]Computer Science [cs]
0101 mathematics
[MATH]Mathematics [math]
NVIDIA GPU
Mathematics
Block (data storage)
Multiply–accumulate operation
Applied Mathematics
[INFO.INFO-NA]Computer Science [cs]/Numerical Analysis [cs.NA]
LU decomposition
Matrix multiplication
rounding error analysis
Computational Mathematics
tensor cores
LU factorization
[INFO.INFO-DC]Computer Science [cs]/Distributed, Parallel, and Cluster Computing [cs.DC]
[MATH.MATH-NA]Mathematics [math]/Numerical Analysis [math.NA]
Subjects
Details
- Language :
- English
- ISSN :
- 10648275
- Database :
- OpenAIRE
- Journal :
- SIAM Journal on Scientific Computing, SIAM Journal on Scientific Computing, Society for Industrial and Applied Mathematics, 2020, 42 (3), pp.C124-C141. ⟨10.1137/19M1289546⟩
- Accession number :
- edsair.doi.dedup.....d4662b96a9900853dc9ab819792bb315