Exploring the Limits of Large Language Models: A Systematic Evaluation of Masked Text Processing Ability through MskQA and MskCal

This paper evaluates Large Language Models’ (LLMs) limitations through their masked text processing capabilities. We introduce two novel tasks: MskQA, which measures reasoning on masked question-answering datasets, and MskCal, which assesses numerical reasoning on masked arithmetic problems. Testing GPT-4o and 4o-mini reveals that LLM performance depends significantly on masking rates and semantic information availability. Our experiments demonstrate that performance decreases as semantic information is reduced, with GPT4o consistently outperforming 4o-mini, particularly in numerical reasoning tasks. The study shows that LLMs maintain reasonable accuracy with masking rates below 40%, but struggle with heavily masked computational tasks. Our findings illuminate the interaction between background knowledge and reasoning ability in masked text processing, highlighting the need for more robust methods to assess LLMs’ true comprehension capabilities.