Methods for improving controlled experimentation in software engineering

Within the past decade Computer Science researchers have begun to use controlled experimentation to address questions of human factors in the programming process. However, some of this work has been criticized for lack of experimental controls, insufficient sample sizes, and questionable generality. In a study of 160 student and professional programmers, several key biographical factors have been identified which account for a large proportion of performance variation on typical experimental tasks. The study has implications for subject selection and interpretation of results in software experimentation. Our findings should lead to improved experimental design and analysis techniques with increased confidence in the results of software psychology studies.