Usability and Utility of Virtual and In-Person Direct Observation of Resident-Patient Encounters

OBJECTIVE Direct observation (DO) of clinical encounters is fundamental in medical education, but barriers to in-person observation of encounters (IOE) have been well described. Virtual observation of encounters (VOE) could be an alternative, but related literature is sparse. This study compared VOE with IOE of pediatric residents in terms of usability, utility, and acceptability. METHODS We conducted a randomized cross-over interventional study of first-year residents in the hospital medicine setting, with each individual undergoing 1 VOE and 1 IOE. Likert-style surveys based on the Technology Acceptance Model, System Usability Scale, and Mini Clinical Evaluation Exercise were presented to the resident, observer, and family after each observation. The resident and observer were surveyed again after completing both observations. Responses were compared using Wilcoxon signed-rank test. RESULTS After 40 encounters, no significant differences were found for time, effort, comfort, or perceived performance effects between VOE and IOE. IOE had higher median ratings for adequate observation of interview, examination, rapport, efficiency, and competence. On a 1-to-7 (“strongly prefer in-person” to “strongly prefer virtual”) scale, median (IQR) response was 2 (2, 5) for observers and residents (P = .86). Technology issues occurred in 60% of VOEs. Observers without technology issues preferred VOE (median [IQR] 5 [4, 6]); observers with technology issues preferred IOE (median [IQR] 2 [1.5, 2]; P < .01). Families indicated comfort regardless of format. CONCLUSIONS This study identified high usability and utility for IOE and VOE from observer, resident, and family perspectives. IOE scored higher for adequacy of observation and overall preference. VOE may be an acceptable alternative when IOE is less feasible, especially with reliable technology.