How to grade the accuracy of the BLAS

Motivated by accelerating machine learning (ML), many computer vendors are building accelerators for matrix multiplication, which save time and energy by operating in lower precisions needed for ML. This has in turn motivated many efforts to use these accelerators to provide faster matrix multiplication implementations with the higher precision required by many other linear algebra applications. Motivated by the large design space of algorithms for approximating higher precision, with significant performance/accuracy tradeoffs, we provide a metric to “grade the accuracy” of a matrix multiplication implementation (or the BLAS more generally), ranging from an “A” for attaining the classic floating point error bound, to a “C” for attaining a weaker but still useful bound, that is satisfied by Strassen-like algorithms. We also propose “ungameable” tests that vendors or users can run to verify their promised accuracy, and describe how these different grades impact the accuracy of applications like Cholesky decomposition. Our test code is publicly available, and we plan to publicly release all our test results. Future work includes expanding our testing and grading to other BLAS routines, nonsquare matrices, use of higher internal precision, complex arithmetic, consistent exception handling, underflow handling, and other high level routines like LU and QR.