Evaluating Arithmetic Reasoning Skills Beyond Benchmark Accuracy: A Session-Based Experimental Design for LLMs

Although LLMs demonstrate high success in logical tasks, it remains debated whether this stems from genuine reasoning or statistical memorization. This study presents a novel, multi-dimensional, perturbation- and session-based behavioral testing framework to assess LLMs’ arithmetic reasoning. To reduce the potential effect of data contamination, texts from the DROP, GSM8k, and MATH datasets were structurally manipulated. A comprehensive analysis was conducted on seven different models (DeepSeek-r1, DeepSeek-v2, Gemma, Llama 3.1, Llama 3.2, Mistral, Qwen) across 90 unique base questions evaluated under 2 distinct format constraints and 4 different presentation scenarios, analyzing a total of 5040 responses. The evaluation combined three quantitative metrics with six qualitative metrics examining session-based vulnerabilities. Results reveal that, within the tested contexts, models suffered performance degradation and logical disconnections when presentation conditions, formats, or constraints changed. Furthermore, models with thinking mechanisms often failed to complete challenging tasks within the time limit rather than gaining an advantage from their reasoning process. Ultimately, these findings indicate that benchmark accuracy alone may not reflect robust reasoning, underscoring the critical need for in-depth evaluation protocols that focus on cognitive consistency and error anatomy rather than mere accuracy rates.