Stage-Aware Optimization of Llama 3 Model Inference for Language Services on Many-Core CPUs

LLM-based language services (e.g., summarization, translation, and question answering) require low-latency and cost-effective inference, yet GPU resources may be costly, unavailable, or difficult to integrate in many deployments. Many-core CPUs provide a flexible alternative, but achieving efficient inference requires optimizations that match the distinct computational characteristics of the Prefill and Decode stages. This paper presents a stage-aware optimization approach for Llama 3 inference on two Phytium many-core processors (FT-2000+/64 and FT-1500A). Fine-grained profiling shows that matrix-vector multiplication (MVM) in the Decode stage accounts for 99.4% of Decode time, whereas Prefill exposes additional token-level parallelism. Accordingly, in Prefill we reorganize multiple token-wise MVM operations into matrix–matrix multiplication and select the GEMM implementation by micro-batch granularity, including a blocked C GEMM (up to 7.8× over token-wise MVM) and a LibShalom-based tile path. In Decode, we apply OpenMP-based intra-kernel parallelism to MVM, RMSNorm, Softmax, and attention-head computation. Experiments on representative GLUE workloads show that the self-kernel implementation improves time per output token (TPOT) by 1.67× at two threads (2.80× at eight). A complete baseline study then shows that a tuned BLAS GEMV accelerates the dominant Decode MVM by a further 3.1×; following the stage-aware selection principle, the final configuration reaches a TPOT of 141.9 ms—a 17.2× end-to-end speedup (7.05 tokens/s per 16-core node)—with scaling that continues to all cores. A kernel-level scalability study on Phytium FT-2000+/64 (Decode MVM up to 8.4× at eight threads) further shows that saturation beyond eight threads is a property of the kernel implementation rather than of the platform. These results demonstrate that stage-aware characterization and per-stage implementation selection can substantially improve single-request Llama 3 inference efficiency on many-core CPUs, and provide guidance for CPU-side optimization in language-service deployments.