Validation of simulation for recruitment to training posts in anaesthesia

Simulation in anaesthesia is increasingly being used for training and assessment of competencies [1], but its role in recruitment is less widely acknowledged. Medical simulation has been used as part of the selection process for training posts in anaesthesia at the South West Peninsula Deanery for the past 2 years. This study examines the face validity, internal reliability and predictive validity of using simulation as part of a selection process. Candidates at the simulation station were scored by two independent assessors. Candidates’ performance was scored on a scale of one (poor) to four (excellent) over each of four separate behavioural traits (e.g. team working ability), giving a maximum score of 16 from each assessor. Correlation of the scores given by the two assessors was used as a measure of internal reliability. At the time of the selection process feedback forms were completed by all candidates and assessors. The simulation station was scored on a five-point scale (one being low and five being high) for each of the following domains: Relevance to selection, Fairness, Opportunity to demonstrate ability, and (for assessors only) Appropriateness to the selection process. All successful candidates were followed up by means of workplace assessments by two assessors on key index cases. The correlation of scores from the simulator station with subsequent workplace assessment scores was calculated to provide a measure of the predictive validity of using simulation in the recruitment process. The inter-assessor correlation score (internal reliability) for candidates’ performance is 0.79. Feedback scores (out of five), expressed as mean (SD) from candidates from the 2008 round of appointments are as follows: Relevance 4.21 (0.91), Fairness 4.05 (0.79), Opportunity 4.14 (0.80). These scores demonstrate the face validity of the simulation station. The correlation of simulation scores with subsequent workplace assessment scores is 0.40 (interim result). These data show that a well constructed simulation scenario is a valid tool in the recruitment to training grades in anaesthesia. High degrees of internal reliability and face validity are demonstrated. A more complete demonstration of validity assessments will be presented. Further measures of predictive validity will also be presented.