Frontiers in Artificial Intelligence Algorithm Optimization: A Comprehensive Review of Training-Time and Inference-Time Advances

The rapid progress of artificial intelligence (AI) has been largely driven by the scaling of deep neural networks, advances in hardware accelerators, and the availability of large-scale datasets. However, the computational, memory, and energy demands of training and deploying foundation models such as GPT-5 and LLaMA-3 have created scalability and sustainability bottlenecks. Algorithmic optimization has emerged as a central strategy to alleviate these challenges across training-time efficiency, inference-time acceleration, long-context extension, and alignment learning. This article provides a comprehensive review of the state of the art in AI algorithm optimization, systematically categorizing approaches, benchmarking them under unified metrics (memory, throughput, latency, perplexity, stability, complexity, portability), and identifying failure modes and boundary conditions. We further present reproducibility artifacts, including minimal training and inference stacks (GaLore + Sophia optimizer; vLLM + FlashAttention-3 + QServe) and standardized datasets (MMLU, GSM8K, LongBench, DCLM). Our synthesis underscores that algorithm–system co-design—spanning optimizer innovations, quantization-aware serving, context length generalization, and efficient preference alignment—is critical to achieving both efficiency and ethical sustainability in next-generation AI systems.