When Cleaning Data Introduces Bias: A Critical Examination of Post-Hoc Methods in Detecting Insufficient Effort Responding

Insufficient Effort Responding (IER) threatens the integrity of survey data, affecting the validity of subsequent analyses. Various post-hoc methods have been developed to detect and eliminate questionable responses, but these methods risk under-detecting IER and over-detecting legitimate responses, introducing analysis biases. This study assessed six such methods—longstring, individual response variability, Mahalanobis distance, psychometric synonym, Standardized Log-likelihood, and Guttman error—as well as their combined application, under diverse conditions. The findings revealed that, in some scenarios, these methods performed no better than random guessing in identifying IER. Further, the performance of combination strategies was merely moderate, falling between the best and worst-performing methods. Alarmingly, excluding responses identified as low-effort often distorted statistical and psychometric estimates more severely than including them. These results underscore the imperative for researchers to judiciously apply IER detection techniques and critically evaluate their effects on data quality.