Cyclical Learning Rate Optimization on Deep Learning Model for Brain Tumor Segmentation

In recent years, deep learning has found widespread applications in tasks like segmentation and classification. Fine-tuning hyperparameters is crucial for improving performance, with learning rate being a key parameter. Various methods, including adaptive learning rates, learning rate scheduling, and cyclical learning rates, have been used to optimize learning rates. Cyclical learning rates offer significant benefits with minimal computational cost, as seen in prior research. This study introduces a novel approach to cyclical learning rate tuning, incorporating Exponential Moving Average. These methods are applied to the BraTS 2021 dataset for segmentation tasks, resulting in superior performance compared to previous approach. Our proposed method reduces the epochs required to reach convergence by 19 and 54 epochs for U-Net and Dense U-net, respectively. For Res U-net, the epoch needed to convergence is 10 epochs more. However, the proposed method produces lower loss values with 0.707, 0.657, and 0.665 compared to previous method with 0.712, 0.685, and 0.725 for U-net, Res U-net, and Dense U-net, respectively.