Key points are not available for this paper at this time.
In online surveys, response time data are often used to make inferences about respondents’ cognitive processing of survey questions and to assess survey data quality. Adequate data preparation is crucial prior to analysis of response time data, in particular the detection and handling of outliers, which are extremely short or long response times. While several outlier detection methods exist, there is little empirical guidance on which method to use and how the choice affects response time data. We compared nine outlier detection methods commonly used in survey research across nine survey questions with varying characteristics, using data from a probability and a nonprobability online panel. Results show substantial differences between outlier detection methods in the proportion of outliers identified and in the effects of outlier exclusion on the response time data, with the effects being more pronounced in the nonprobability panel. Moreover, outlier detection methods differ systematically in the types of cases they classify as outliers, particularly with respect to respondent age and education. Based on these findings, recommendations for outlier detections methods in survey research are discussed.
Hadler et al. (Tue,) studied this question.