Ensuring competence of nephrology fellows performing kidney biopsies using a simulation-based test: Preliminary validity evidence

Background Simulation-based training provides a patient-safe environment for practicing percutaneous kidney biopsies. We aimed to develop and gather preliminary validity evidence for a simulation-based test capable of ensuring basic competency in the performance of kidney biopsies. Methods A procedure-specific assessment tool was developed by three experts, guided by clinical guidelines and an existing generic tool. Novice and experienced physicians were recruited to perform two ultrasound-guided kidney biopsies on a commercially available phantom. All procedures were video-recorded and subsequently scored blindly and independently by two expert raters using the newly developed tool. Results The assessment tool consists of twelve items scored from 1 to 5, i.e., maximum possible score of 60. We found an excellent internal consistency reliability, Cronbach's α = 0.92, and found strong interrater and test-retest reliabilities, Pearson's r = 0.84 and 0.95, respectively. Generalizability theory analyses revealed that three raters assessing one procedure, or two raters assessing two procedures, would result in an overall reliability higher than the recommended threshold of 0.8. Experienced physicians scored significantly higher than novices, with respective mean score of 53.3 (SD 3.8) points and 39.0 (SD 4.3) points, P < 0.001. The established pass/fail score of 47.8 points resulted in one false negative (i.e., one experienced doctor failing the test) and no false positives (i.e., no novices passing the test). Conclusions We developed the first procedure-specific assessment tool for percutaneous kidney biopsy that provides preliminary validity evidence and a credible pass/fail standard for simulation-based training. This tool contributes to patient-safe simulation-based mastery learning where all trainees practice to a predefined proficiency level before being allowed to proceed to supervised practice on patients. While these findings support the use of the tool for formative simulation-based training, further research is needed to establish its predictive validity for clinical performance and patient outcomes.