Short answer questions in an undergraduate qualifying examination: a study of examiner variability

Scripts of a short answer Final MB examination paper were re-marked by the original examiner and three colleagues. While individual question marks varied considerably, most examiners agreed quite well as to the overall rank order of candidates--although not about absolute total marks--but a few disagreed considerably. Examiners exhibited remarkably consistent idiosyncratic behaviour in terms of the distribution of marks awarded. These factors led to haphazard pass/fail and pass/distinction decisions on this particular component of the examination. Recommendations are put forward to improve the marking consistency and use of short answer question papers.