Evaluation and Assessment of the ABATE Framework to Enhance Implicit Bias Training for Virtual Interviews in Medical Schools

OPEN ACCESSJune 28, 2024Evaluation and Assessment of the ABATE Framework to Enhance Implicit Bias Training for Virtual Interviews in Medical Schools Veda Nagubandi1, Caleb C. McKinney, PhD, MPS2, Nicole M. Houle, EdD3, Kristi D. Graves, PhD4, Susan M. Cheng, EdLD, MPP5,* Veda Nagubandi1 Research Fellow, Office of Diversity, Equity, Inclusion, & Belonging, Georgetown University School of Medicine , Caleb C. McKinney, PhD, MPS2 https://orcid.org/0000-0001-7573-3899 Associate Professor, Department of Rehabilitation Medicine, Associate Dean of Graduate and Postdoctoral Training & Development, Biomedical Graduate Education, and Assistant Vice President of Master's Program Administration & Development, Georgetown University Medical Center , Nicole M. Houle, EdD3 https://orcid.org/0009-0002-3868-7875 Associate Dean for Admissions and Financial Aid, Office of Admissions and Financial Aid, Georgetown University School of Medicine , Kristi D. Graves, PhD4 https://orcid.org/0000-0001-8808-1631 Associate Professor of Oncology, Department of Oncology, and Associate Director of Diversity, Equity and Inclusion, Lombardi Comprehensive Cancer Center, Georgetown University School of Medicine , Susan M. Cheng, EdLD, MPP5,* https://orcid.org/0000-0001-5536-0802 Associate Professor, Department of Family Medicine, and Senior Associate Dean for Diversity, Equity, Inclusion, & Belonging, Office of Diversity, Equity, Inclusion, & Belonging, Georgetown University School of Medicine E-mail Address: [email protected] Author Information Veda Nagubandi1, Caleb C. McKinney, PhD, MPS2, Nicole M. Houle, EdD3, Kristi D. Graves, PhD4, Susan M. Cheng, EdLD, MPP5,* 1 Research Fellow, Office of Diversity, Equity, Inclusion, & Belonging, Georgetown University School of Medicine 2 Associate Professor, Department of Rehabilitation Medicine, Associate Dean of Graduate and Postdoctoral Training & Development, Biomedical Graduate Education, and Assistant Vice President of Master's Program Administration & Development, Georgetown University Medical Center 3 Associate Dean for Admissions and Financial Aid, Office of Admissions and Financial Aid, Georgetown University School of Medicine 4 Associate Professor of Oncology, Department of Oncology, and Associate Director of Diversity, Equity and Inclusion, Lombardi Comprehensive Cancer Center, Georgetown University School of Medicine 5 Associate Professor, Department of Family Medicine, and Senior Associate Dean for Diversity, Equity, Inclusion, & Belonging, Office of Diversity, Equity, Inclusion, & Belonging, Georgetown University School of Medicine ∗Corresponding author: [email protected] https://doi.org/10.15766/mep_2374-8265.11416 SectionsAboutPDF ToolsDownload Citations ShareFacebookXEmail Abstract Introduction: The influence of implicit biases in virtual interviews must be addressed to ensure equity within the admissions process. ABATE is a mnemonic framework of five specific categories of implicit bias (affinity-based, backdrop-based, appearance-based, technology and media-based, and enunciation-based biases) that should be anticipated and mitigated for faculty, staff, health professionals, and medical students who conduct virtual interviews at medical schools. Methods: A 60-minute workshop was developed to educate medical school admissions interviewers about the ABATE model and strategies to mitigate implicit bias during virtual interviews. Four workshops were held over 1 year totaling 217 individual attendees. The workshops were evaluated using a single-group, pre-post questionnaire designed with the Kirkpatrick evaluation model. Results: Attendees reported that they found the ABATE workshop useful and relevant to improving their ability to minimize implicit bias during virtual interviews. Significant improvements were found in attendee reactions to the utility of implicit bias training (M pre = 2.6, M post = 3.1, p = .002). Significant changes were also reported in attendees' attitudes about interviewing confidence (M pre = 3.0, M post = 3.2, p = .04), bias awareness (M pre = 3.0, M post = 3.4, p = .002), and identifying and applying bias mitigation solutions (M pre = 2.5, M post = 3.0, p = .003). Knowledge specific to backdrop-based biases also significantly increased (M pre = 3.2, M post = 3.4, p = .04). Discussion: The ABATE workshop demonstrates promise in mitigating implicit bias in virtual medical school interviews. Educational Objectives By the end of this activity, learners will be able to: 1.Identify common interview biases.2.Apply the ABATE model to interview scenario examples, translating strategies from in-person interviews to virtual contexts.3.Facilitate equitable and inclusive virtual interview processes, especially for candidates underrepresented in medicine. Introduction Implicit biases are the positive and negative attitudes or stereotypes that can unconsciously influence people's perception, actions, and decisions.1 Consistent evidence demonstrates that implicit bias influences the recruitment, interviewing, assessment, and selection of applicants and candidates for positions.2 Everyone is susceptible to implicit biases, which can be activated involuntarily, without individuals realizing their presence or influence on decisions.3,4 Interviewing is an interpersonal process in which subjective impressions can drive decisions for admissions and job selection.5,6 Although all individuals hold implicit biases, everyone, including interviewers, can be susceptible to illusions of objectivity. When people believe they are objective, they may act on biases more frequently than individuals who acknowledge the subjective influence of initial impressions.7,8 Medical education studies have demonstrated that interviewer bias affects how candidates' interviews are scored.9 In a study examining medical school admissions interviews in Canada, interviewer mood impacted candidate ratings; candidates who interviewed on rainy days tended to receive a 1% lower score than others who interviewed on sunny days, controlling for the applicants' qualifications.10 Contextual elements such as time pressure, cognitive load, and inattentiveness to tasks contribute to an environment where unconscious bias can be activated.11 Fortunately, behavioral responses from implicit biases can be attenuated once interviewers gain awareness of their biases.3,12 In May 2020, the Association of American Medical Colleges (AAMC) advised medical schools to adopt either phone or online interviews, and the Coalition for Physician Accountability recommended that all interviews be conducted virtually for the upcoming application season.13,14 The COVID-19 pandemic marked a "landmark change in the way [graduate medical education] selection will occur and may irrevocably change the process going forward."15 Institutions embraced a sudden change to virtual interviewing formats given the decreased risks and costs for candidates and institutions.16 A recent study indicated that although both medical students and residents prefer in-person interviews, both groups also point to the merits of virtual interviews as a practical option, given the benefits associated with reduced costs and time.17 In May 2023, the AAMC recommended that academic medical programs continue to employ virtual interview formats to improve equity and decrease environmental impact.18 With virtual interviews remaining highly relevant, the influence of implicit biases during virtual interviews must be addressed to ensure equity in the interviewing process. These implicit biases can disproportionally and negatively affect applicants who are underrepresented in medicine. Although the pervasiveness and adverse impact of implicit bias during in-person interviews are well documented, less is known about how bias manifests during virtual interviews and how best to mitigate its effects. To address this urgent need, author Susan M. Cheng developed ABATE, a practical framework to guide equitable virtual interview and selection processes. ABATE is a mnemonic framework for five specific categories of implicit bias that should be anticipated and abated during interview processes, with attention to virtual settings: •Affinity-based biases: in-group affinity based on likeness and cultural matching.19•Backdrop-based biases: subtle cues based on background decor/screens found in the physical environment.20•Appearance-based biases: attractiveness based on outward assessments of readily visible factors including weight, race and ethnicity, and perceived beauty.21•Technology and media-based biases: technology challenges (i.e., limited access to internet, need for high-quality cameras and speakers, mental fatigue caused by the cognitive demands associated with virtual conference platforms like Zoom) can exacerbate biases and the potential for racial bias as certain aspects of video camera functioning (e.g., clarity of images, automatic adjustment of lighting) have been programmed based on individuals with lighter skin. Additionally, the move from 3D (in-person) to 2D (virtual) makes it more difficult to fully convey information, leading to interviewers and applicants viewing each other less favorably.22–24•Enunciation or vocal cue-based biases: impression management can be swayed by vocal cues including tone, accent, pitch, and frequency.25,26 We developed a workshop to educate medical school admissions interviewers about the ABATE model and strategies to mitigate implicit bias during virtual interviews. Methods Four workshops were delivered between August 2021 (two workshops) and August 2022 (two workshops). We evaluated the effectiveness of the ABATE model using a single-group, pre-post test design, measuring attendees' levels of knowledge acquisition and changes in attitudes before and after attending the workshop. We obtained approval from Georgetown University's Institutional Review Board to perform retrospective analysis of workshop evaluation materials (STUDY00004672). Context of Prior Workshops for Admissions Interviewers Annual workshops for admissions interviewers have been required and the content of these preparatory workshops has evolved over the years. In prior years (before 2016), the workshop outlined the admissions process and described admissions requirements and guidelines for competitiveness set forth by the selection committee. It also covered interview etiquette and provided requirements to interviewers about what questions to ask the applicants and what questions to avoid. Finally, the workshop delivered instructions on writing a comprehensive evaluation of an applicant's noncognitive skills (e.g., soft skills related to motivation, integrity, and interpersonal interactions).27 Starting in 2016, the Office of Admissions expanded the interviewer workshop by including implicit bias training. When the COVID-19 pandemic forced the interview process to go virtual, the implicit bias training was adapted to the 60-minute ABATE workshop to instruct interviewers on ways to identify and mitigate implicit biases during virtual interviews. Preparation ABATE development was guided, in part, by Kern's six-step approach28: •Step 1. Problem Identification and General Needs Assessment: consulted with admissions personnel and leadership during the start of the COVID-19 pandemic to assess the need for guidance on mitigating potential biases arising from the quick pivot to online interviews.•Step 2. Targeted Needs Assessment: conducted a literature review to assemble evidence-based bias mitigation strategies into ABATE framework components.•Step 3. Goals and Objectives: developed the following goals in conjunction with admissions personnel and staff at the Office of Diversity, Equity, Inclusion, & Belonging to update the already-required anti-racism training for the virtual interview context: ○Evaluate the feasibility of incorporating the ABATE framework in interviewer training,○Measure current interviewer knowledge of issues surrounding implicit bias, and○Evaluate an ABATE workshop on interviewer reactions to the training, attitudes about implicit bias, and knowledge.•Steps 4 and 5. Educational Strategies and Implementation: incorporated framework into an interactive and conversational style workshop that was made mandatory for all admissions interviewers by the Office of Admissions in August and September before the admissions cycle.•Step 6. Evaluation and Feedback: collected feedback from pre- and postworkshop questionnaires and verbally from faculty and staff through follow-up assessments and conversations with the admissions team.29 Following development of the ABATE model and its supporting elements (the framework and related ABATE model materials: Appendices A and B), we developed pre- and postworkshop evaluation questionnaires (Appendices C and D). Equipment The workshops were delivered virtually through the Zoom audiovisual platform, and the ABATE workshop slide deck (Appendix E: slides, Appendix F: speaker notes) was shared through screen-sharing capabilities. All medical school admissions interviewers who attended the workshop needed a working computer or other device with an audiovisual system to participate and complete the pre- and postworkshop evaluation questionnaires. Personnel The 60-minute workshop included a designated question-and-answer period and was delivered by one facilitator with expertise in both bias mitigation and educational session delivery for faculty and staff using online platforms (e.g., Zoom). ABATE Workshop: Procedures and Content At the start of the workshop, attendees were given 5 minutes to complete an online anonymous preworkshop questionnaire. It should be noted that responding to the questionnaire was not required to continue to the next stage of the workshop. In Part 1 (15 minutes) of the ABATE workshop, the facilitator described and defined types of bias, presented facts regarding obstacles to achieving diversity, and set the stage for the importance of mitigating implicit bias within virtual admissions interviews. This introduction also included examples of common interview biases in the context of virtual interviews (Appendix A) and examples of ABATE framework application at the three levels described as follows: (1) institutional (ABATE incorporated into training for selection committees responsible for admissions, appointments and promotions, and recruitment), (2) interviewer (ABATE incorporated as part of structured training and as part of a checklist to review in a just-in-time manner before interviewing), and (3) interviewee (ABATE distilled into a best-practices guidance document or webinar slides to prepare for interviews and neutralize the impact of implicit bias in ways that candidates can control; Appendix B). The facilitator encouraged participation by inviting attendees to identify types of biases evident in sample interviewer notes; attendees shared their ideas and thoughts by using either the chat or the audio functions in the Zoom platform. In Part 2 (15 minutes) of the ABATE workshop, the facilitator discussed the structure of the ABATE model with example interview scenarios encapsulating each of the five types of implicit bias represented and encouraged participation by inviting attendees to identify types of biases involved in sample interviewer notes. In Part 3 (15 minutes), the facilitator provided examples of bias mitigation from the ABATE model implementation materials. Appendix materials were made accessible for attendees by uploading them to the resources tab on the software system (AMP by ZAP Solutions) that interviewers used to complete their assessments of interviewees. For interviewers who lacked access to AMP, files were also uploaded to the university's secure file-sharing server, Box, to ensure accessibility. Debriefing After parts 1–3 of the ABATE workshop, the facilitator encouraged interviewers to consider the material presented and share their reflections and questions for 5 minutes. The facilitator fostered an open discussion about attendee experiences while incorporating structured suggestions for individual improvement based on responses provided by the attendees to mitigate instances of implicit bias during virtual interviews. Following the debrief discussion, the facilitator directed attendees to take 5 minutes to complete an anonymous postworkshop questionnaire through a link provided in the Zoom chat. Similar to the preworkshop questionnaire, responses to the postworkshop questionnaire were not required. Course Evaluation Our evaluation followed the Kirkpatrick evaluation model, an empirically based approach used in academic curriculum development to evaluate training programs.30 Level 1 of the Kirkpatrick evaluation model assesses attendee reactions regarding the utility of training programs, while level 2 assesses improvement in attendee expertise, knowledge, attitudes, and/or skill because of the training program. Preworkshop questionnaire The attendees completed a nine-item preworkshop questionnaire to assess baseline reactions to the utility of bias mitigation training programs, attitudes, and knowledge levels. All responses to the nine items were made using a 5-point Likert scale (strongly agree, agree, neutral, disagree, and strongly disagree). For level 1 of the Kirkpatrick evaluation model, one question assessed attendee reactions regarding the value of training programs (question 1 pertaining to Educational Objective 3; see Table 1). For level 2 of the Kirkpatrick evaluation model, three questions assessed self-reported attendee attitudes surrounding confidence in mitigating implicit bias and recognizing sources of implicit bias (questions 2–4 pertaining to Educational Objective 3; see Table 1), and five optional questions gauged baseline knowledge surrounding implicit bias recognition of the five tenets of the ABATE model (i.e., affinity-based, backdrop-based, appearance-based, technology and media-based, and enunciation or vocal cue-based biases; questions 5, 7, and 9 pertaining to Educational Objective 1; questions 6 and 8 pertaining to Educational Objective 2; see Table 1). It should be noted that other interpretations of the Kirkpatrick evaluation model could place attitude questions 2–4 under level 1 given that attitudinal responses about one's knowledge could be considered part of the reaction to training. Table 1. Statistical Results for Pre- and Postworkshop QuestionnairesTable 1. Statistical Results for Pre- and Postworkshop Questionnaires Postworkshop questionnaire The postworkshop questionnaire consisted of the same nine questions as on the preworkshop questionnaire to assess attendees' reactions, attitudes, and knowledge. An open-ended question was also included for attendees to provide additional comments or suggestions for improvements in future ABATE workshops. Results The workshop was offered twice in 2021 (130 attendees; 59 completed the preworkshop questionnaire [45% response rate], and 26 completed the postworkshop questionnaire [20% response rate]) and twice again in 2022 (128 attendees; 16 completed the preworkshop questionnaire [12% response rate], and 15 completed the postworkshop questionnaire [12% response rate]). Across all workshops, we had a total of 258 attendees, of whom 41 attended the workshop twice (resulting in 217 unique attendees). For the 2021 admissions cycle, the Georgetown University School of Medicine (GUSOM) had 132 volunteer interviewers, of whom 38% had no prior experience with interviewing for the Office of Admissions. Faculty and staff constituted 71% of the group. Interviewers' self-reported gender identities were 56% female and 44% male, with no individuals selecting other gender identity categories. In the 2022 admissions cycle, GUSOM had 142 volunteer interviewers, of whom 60% were faculty and staff and 32% were new to interviewing. Overall self-identified genders were 43% female and 57% male, with no individuals selecting other categories. Given the medical school's mission to support an equitable admissions process, ABATE workshop attendance was required by the Office of Admissions for both the 2021 and 2022 interview cycles as a prerequisite for conducting interviews. Results from all workshop questionnaire responses were aggregated. Likert-scale responses ranged from 0 (strongly disagree) to 4 (strongly agree). A few items were reverse scored (see note for Table 1). In analyses conducted to generate Figures 1 and 2, we combined responses of strongly agreed and agreed into one category and also combined responses of strongly disagreed and disagreed. Responses of neutral remained as their own category. These three categories were not assigned numerical values. Pre- and postworkshop questionnaire responses were not matched among participants. We used t tests to calculate differences between the pre- and postworkshop scores and calculated variances to determine use of appropriate t-test statistics based on whether the distribution indicated an equal or unequal variance. Figure 1. Pre- and postworkshop questionnaire results for level 1 (reaction) of the Kirkpatrick evaluation model. Question read as follows: "My prior training has adequately prepared me to minimize implicit bias during virtual interviews." Responses that marked strongly agree or agree were aggregated and considered positive, while responses that marked strongly disagree or disagree were aggregated and considered negative. Refer to Table 1 for exact means, standard deviations, and significance values of questionnaire responses (n values are indicated on the figure). Figure 2. Pre- and postworkshop questionnaire results for level 2 (attitude and knowledge) of the Kirkpatrick evaluation model. Attitude-measuring questions included the following: (1) interviewing confidence statement: "I am confident in my abilities to conduct an unbiased virtual interview"; (2) bias awareness statement: "I am comfortable with saying that I am aware of any potential sources of implicit bias that may occur during virtual interviews"; and (3) solution identification/application statement: "I have experience with applying and identifying solutions that reduce implicit bias during my virtual interviews." Responses that marked strongly agree or agree were aggregated and considered positive, while responses that marked strongly disagree or disagree were aggregated and considered negative. Knowledge-measuring questions began with the stem "Based on your knowledge of the ABATE model for identifying bias please assess this statement" in the postworkshop questionnaire only; the question stem was absent in the preworkshop questionnaire. The question statements included the following: (4) affinity biases: "Generally, an interviewer's perception of 'likeness,' or membership to an 'ingroup' puts candidates at an advantage"; (5) backdrop biases: "To address the potential undue influence of interview backgrounds on affinity bias and cultural matching, admissions offices and interview teams can identify strategies to neutralize the ways in which the background contributes to implicit bias"; (6) appearance biases: "A candidate that is overweight doesn't understand what it means to be a leader and will not be as successful in medical school than other candidates who are not overweight"; (7) technology and media biases: "Since we've all had to adjust virtually during the pandemic, there is no bias surrounding an interviewee's ability to make eye contact during virtual interviews"; and (8) enunciation or vocal cue biases: "The way one enunciates and pauses in their speaking has nothing to do with how you assess them." Responses that marked strongly agree or agree were aggregated and considered positive, while responses that marked strongly disagree or disagree were aggregated and considered negative for statements 4 and 5. Responses that marked strongly agree or agree were aggregated and considered negative, while responses that marked strongly disagree or disagree were aggregated and considered positive for statements 6–8. Please refer to Table 1 for exact means, standard deviations, and significance values of questionnaire responses (n values are indicated on the figure). Overall, the attendees reported that they found the ABATE workshop useful and relevant to improving their ability to minimize implicit bias during virtual interviews. Prior to completing the ABATE workshop, 40% of respondents either disagreed or had a neutral response whereas 60% of respondents agreed with the statement "My prior training has adequately prepared me to minimize implicit bias during virtual interviews" (question 1). After attending the workshop, no respondents disagreed with the statement, 15% had a neutral response, and 85% agreed (Figure 1). These pre- to postworkshop response shifts were statistically significant (M pre = 2.6, M post = 3.1, p = .002). We also found statistically significant changes in attendees' attitudes about interviewing confidence (question 2: M pre = 3.0, M post = 3.2, p = .04), bias awareness (question 3: M pre = 3.0, M post = 3.4, p = .002), and solution identification and application (question 4: M pre = 2.5, M post = 3.0, p = .003). These results indicate improvements in interviewers' self-reported abilities to confidently conduct an unbiased virtual interview, recognize sources of implicit bias during virtual interviews, and apply mitigating solutions (Table 1 and Figure 2). Notably, changes in responses to the attitude question 4 ("I have experience with applying and identifying solutions that reduce implicit bias during my virtual interviews") reflected improvement in attendee confidence to mitigate biases that might arise during their virtual interviews after attending the workshop. Before attending the workshop, 23% of respondents disagreed with the statement whereas 57% of respondents agreed. After attending the workshop, only 2% of respondents disagreed with the statement while 83% of attendees agreed (Figure 2). Of the five knowledge questions, the item assessing backdrop-based biases showed significant improvement from pre- to postworkshop (question 6: M pre = 3.2, M post = 3.4, p = .04; Figure 2). These results reinforce the role of admissions offices and interview teams in mitigating the effects of implicit biases based on candidates' backgrounds during virtual interviews. We did not find any other statistically significant changes from pre- to postworkshop responses in the other four knowledge questions related to affinity-based (question 5: M pre = 2.8, M post = 2.8, p = .88), appearance-based (question 7: M pre = 3.8, M post = 3.6, p = .48), technology and media-based (question 8: M pre = 2.8, M post = 2.8, p = .86), and enunciation or vocal cue-based biases (question 9: M pre = 2.0, M post = 1.9, p = .62; see Table 1), potentially indicating lack of knowledge acquisition regarding these tenets of the ABATE model. An open-ended question in the postworkshop questionnaire elicited attendees' takeaways, feedback, and comments. Attendee responses were indicative of interviewers' awareness of the pervasiveness of implicit biases in virtual admissions interviews, strong takeaways regarding bias mitigation techniques, and constructive feedback for improving future workshops. Example quotes are included in Table 2. Table 2. Thematic Analysis of Open-Ended Responses on the Postworkshop QuestionnaireTable 2. Thematic Analysis of Open-Ended Responses on the Postworkshop Questionnaire Discussion Bias, if unchecked, threatens the ability to fairly evaluate each candidate. If medical schools aim to uphold an equitable process that assesses the full abilities of all candidates holistically and equally, then virtual interviews require diligent and thoughtful design of new procedures. Despite the easing of pandemic restrictions, several medical schools have announced that interviews for future cycles will be held virtually. The current shift towards virtual interviews increases the need for the design and implementation of empirically supported bias mitigation tools that improve and reimagine what admissions offices can do to create more equitable and bias-free interview processes. These approaches are particularly important as the pool of candidates in medicine and health care continues to diversify. Fundamentally, the greatest promise of the ABATE framework is in its focus on applying bias reduction techniques and promoting greater equity in the virtual assessment of candidates. The ABATE framework identifies five areas of implicit bias with potentially significant influence. Training to identify and rectify these biases can improve decisions in both in-person and virtual interviews for medical school admissions by mitigating the impact of implicit bias. Results suggest that the ABATE training workshops for medical school admissions interviewers are relevant and increase interviewers' confidence and knowledge in identifying and rectifying sources of implicit bias in virtual interviews. The positive shift in attitudes seen among interviewers after the ABATE workshop is particularly promising, and future work can examine the effects of ABATE training on long-term behavioral changes. It is important to note that our institution has made a greater effort over recent years to shift institutional culture as we continue to develop and offer extensive training related to diversity, equity, and inclusion. Since personnel are exposed to other thematically similar training sessions before attending ABATE workshops, this could have influenced the positive shifts seen in reactions and attitudes surrounding ABATE training. Limitations of this study include low questionnaire response rates for all sessions, with notable drop-off for the postworkshop questionnaire for the 2021 sessions. The low response rate can likely be attributed to the nonmandatory nature of the questionnaires. We