Silent Numerical Failures in On-Device ML Converters: A Systematic Audit of FP16 Overflow in Apple Neural Engine Deployment

Key Points

• This work aims to audit the performance of FP16 arithmetic in Apple's Neural Engine to identify and address silent numerical failures.
• Conducted a systematic audit of coremltools converter in Apple's Neural Engine.
• Identified five specific operations with overflow issues during computation: softplus, mish, reduce_log_sum_exp, log_softmax, and logcumsumexp.
• Developed and implemented numerically stable alternatives and conducted regression tests.
• Five operations were found to silently yield incorrect results on ANE due to FP16 overflow.
• Proposed numerically stable versions of operations were developed, enhancing the reliability of the coremltools.
• The findings are critical for key AI models like YOLO, BERT, ViT, and MobileNetV3, affecting their deployment on Apple devices.