Usability Testing vs. Heuristic Evaluation: A Head-to-Head Comparison

The importance of user testing, heuristic evaluation and iterative design in the development of computer software programs was examined. In the first study, twenty-five subjects with limited computer experience, were randomly divided into five groups of five subjects each. All groups were asked to perform a telephone bill inquiry task using two character-based screens. After having one group perform, one change per screen was made before beginning the testing of the next group. The system was improved three times. A final experimental group completed the same task using an “ideal” system designed and presented by Molich and Nielsen (1990). Rather than the 29 changes originally suggested by Molich and Nielsen, our results showed that only one change to each of the original screens was necessary to achieve the same performance and preference levels as those demonstrated by their “ideal” system. The same task was repeated using a graphical user interface. A heuristic evaluation suggested up to 43 potential changes, whereas the usability test demonstrated that only two changes optimized performance. These findings demonstrate one of the major weaknesses of heuristic evaluations, and the importance of usability testing in the design and development of human interfaces.