Abstract Objectives: To make (i) a systematic assessment of activation functions for solving initial value problems (IVPs) and ii) the use of Automatic Differentiation (AD) for accurate gradient evaluation in Gradient Descent (GD). Method: Unlike conventional backpropagation approaches that rely on manually derived gradients, the proposed AD-based framework enables precise derivative evaluation using the chain rule, improving numerical stability and convergence efficiency. The performance of the proposed method is validated through several benchmark higher-order IVP test problems and compared with existing numerical techniques including Regression-Based Weight initialization (RBW), Chebyshev Wavelet Collocation Method (CWCM), and Haar Wavelet Collocation Method (HWCM). Findings: In the present study, an enhanced ANN-based methodology incorporating Automatic Differentiation (AD) is developed to solve higher-order initial value problems (IVPs) with improved accuracy and convergence. The numerical experiments indicate that the proposed ANN-AD approach achieves maximum absolute errors, depending on the order and complexity of the test problems. In several benchmark cases, the method consistently attains errors below 10−6, demonstrating superior approximation accuracy compared to existing ANN-based approaches reported in the literature. Furthermore, the proposed method exhibits stable convergence behaviour across first, second and fifth-order IVPs without requiring problem-specific tuning. These results confirm that the proposed methodology provides an accurate, flexible and computationally efficient framework for solving higher-order differential equations. Additionally, its robustness and generalization capability suggest strong potential for extension to more complex boundary value problems and nonlinear differential systems in future work. Novelty: This work combines Automatic Differentiation– driven gradient optimization with activation-function sensitivity analysis in a unified ANN framework for higher-order IVPs, a combination not systematically explored previously. Keywords: Differential equations; Neural network; Activation function; Automatic differentiation; Gradient descent
Jayasree et al. (Wed,) studied this question.