Key points are not available for this paper at this time.
We present a detailed case study of a computer scientist learning and using the Cognitive Walkthrough (CW) technique to assess a multimedia authoring tool. We then compare the predictions produced by the analysis to the usability problems actually found in empirical usability tests. In this case, the concepts of Cognitive Walkthrough were learnable by the analyst who had little prior training in psychology or Human-Computer Interaction, flexible enough for the analyst to make reasonable modifications to the technique to fit the design situation, but disappointingly ineffective in predicting actual user problems. We present several hypotheses about the cause of low effectiveness, which suggest that additional knowledge, currently tacit in CW-experts, could be used to improve the technique. In addition, the emergent picture of the process this evaluator went through to produce his analysis sets realistic expectations for other novice analysts who contemplate learning and using Cognitive Walkthroughs.
John et al. (Sat,) studied this question.