Shoulder surfing experiments: A systematic literature review

In search of the silver bullet to solve the password problem, the field of knowledge-based authentication has become bloated with novel proposals aiming to replace textual passwords. The emphasis on the quantity of studies as opposed to the quality of evaluation has made it difficult to compare the methods, as well as to validate and generalize the results. To improve the quality of security and usability evaluations, experimental design decisions should be reviewed and standardized. In this systematic review, we focus on the evaluation of the shoulder surfing attack (SSA) vulnerability. We formulate two research questions to help us determine how the design of the method should affect the SSA experimental design process, and how different design decisions affect the validity and interpretability of the results under various assumptions and threat models. To provide the researchers with comprehensive literature on SSA evaluation, we identify empirical shoulder surfing studies conforming to a predefined set of quality criteria. Based on the design features extracted from the experiments, we develop an evaluation framework for the assessment of the shoulder surfing experimental setup. In the follow-up analysis, we assess the proposed methods’ design features, and the quality of their SSA experiments, using Schaub et al.’s design aspect and our SSA evaluation frameworks, respectively. Through exhaustive analysis, we strive to streamline and standardize experimental decisions by showcasing their impact on the outcome of the study, and generate guidelines for a more objective design of shoulder surfing experiments.