The Validity and Reliability of Oral Examinations in Assessing Cognitive Skills in Medicine1

In order to attempt to assess aspects of clinical competence, not adequately assessed by other means, the Center for the Study of Medical Education, University of Illinois College of Medicine together with the American Board of Orthopaedic Surgery developed oral examinations in formats specifically designed to yield information on high level cognitive functioning. The examinations were administered to 784 candidates for certification in January 1968. Reliability of the oral problem‐solving component score pooled from four examiners was approximately .50. Assessment of content, construct, and concurrent validity made by questionnaire and factor analytic studies indicated that the oral tests identified factors not measured by multiple‐choice tests and, therefore, significantly improved the relationship between supervisory evaluations and test scores.