Methodical Investigation of Loss Functions in Deep Learning and Machine Learning

Deep learning (DL) is applied for solving a wide range of complex tasks such as, image recognition, natural language processing, segmentation, autonomous driving, and healthcare diagnostics. Its performance largely depends on loss functions, which measure deviation between predicted and actual values during training. So, selecting an appropriate loss function (also called cost-function) improves the success rate. Over the past few decades, researchers have developed many loss functions for ML and DL applications. Novel loss functions address issues such as imbalanced data, uncertain distributions, bounding box overlaps, and overfitting. In this paper, a comprehensive review of widely used 72 loss functions in ML and DL is discussed along with their pros and cons, mathematical representations, and limitations. They are also grouped based on their nature and applications. These loss functions were mainly categorized into classification, regression, ranking, margin based, GAN, and segmentation. This review paper incorporates experimental validation on paroxysmal atrial fibrillation ECG data to substantiate the discussion. A thorough analysis of traditional and recent loss functions is included to highlight their relative strengths in practical applications. This integrated perspective blends theoretical ideas and empirical evaluation, providing readers an in-depth understanding of current progress and future directions in the field of study.