The Program Self-Study and the 10-Year Site Visit: Rationale for a New Approach

When the leadership of the Accreditation Council for Graduate Medical Education (ACGME) conceptualized what came to be known as the Next Accreditation System, a key component was a comprehensive self-study for programs and sponsoring institutions. The intent for the self-study, and for the overall approach to accreditation, was to promote improvement and recognize that most programs already comply with the vast majority of standards. This required a self-study that would allow programs to set aspirational aims, to analyze community needs, and to initiate activities to further these aims, to facilitate improvement in areas where the program already complied with the accreditation standards. This focus on ongoing improvement, through the annual program evaluation and a more formal self-examination every 10 years, was intended to make the accreditation process more relevant and meaningful for the majority of programs already in substantial compliance, and to convert the national graduate medical education “enterprise” into one that seeks excellence beyond satisfaction of minimum standards.The intent was a comprehensive self-study with a subsequent site visit that would be based on “a description of how the program [or sponsoring institution] creates an effective learning and working environment, and how this leads to desired educational outcomes, and an analysis of strengths, weaknesses, and plans for improvement.”1 The ACGME’s initial plans for the self-study called for a review of the findings during the program’s 10-year site visit and an assessment during that site visit “that the self-study document offers an objective, factual description of the learning and working environment.”1During late winter and spring of academic year 2014–2015, as the dates for the first program self-studies and associated site visits neared, ACGME leadership undertook a comprehensive reexamination of the proposed approach to the self-study. This was done in response to input from the graduate medical education community, which suggested that the review of the self-study during an accreditation site visit might create a barrier to a frank assessment. It also became clear that the timing initially envisioned, with the self-study being followed closely by an accreditation site visit, would not allow programs to make improvements in areas identified in the self-study. This would reduce the effectiveness of the self-study in facilitating program improvement, as well as the ability of the new accreditation system to document ongoing improvements in all programs to meet the ACGME goal of public accountability for the quality of physician education.In response to the concerns voiced by the community, a new approach to the self-study and the 10-year site visit was developed in an iterative fashion, with input from experts, the community, and ACGME governance. It has 4 components, shown in the box. They are described in more detail below, along with the rationale for each element.The first component seeks to protect the information the program collects and uses in its self-assessment and self-improvement process from use in program accreditation. The rationale is to preserve the intent for the self-study as an introspective deliberation on the program’s aims, the context in which it operates and the community it serves, and how the program improves and innovates in ways that are relevant to the aims, context, and community. The chosen approach ensures that information on program areas of improvement is not disclosed before or during the 10-year site visit. This change was made to address concerns that program leaders might be less willing to identify and explore areas for improvement out of concern that this information, if presented during an accreditation site visit, may result in citations by the Review Committee.The second component is a temporal separation to give programs time to make and demonstrate improvements in areas identified during the self-study. The 12- to 18-month interval between the self-study and the 10-year site visit is intended to allow programs to implement improvements in areas identified during the self-study prior to undergoing an accreditation site visit.The third element is a pilot test of an added, voluntary, nonaccreditation site visit with feedback. The aim is to assess whether this added visit and feedback, soon after the program has conducted its self-study, will accelerate improvement. The pilot is open to core programs in the 7 Phase I specialties (emergency medicine, internal medicine, neurological surgery, orthopaedic surgery, pediatrics, diagnostic radiology, and urology) if their initial site visit is scheduled between April 2015 and January 2017. The added site visit will occur 1 to 3 months after the program has completed its self-study and will be conducted by a team of 2 field representatives who have received specific training in this role. If the core programs volunteer to be part of this pilot, subspecialty programs may opt in or out of the pilot. After the visit, feedback will be shared with the program. No information gleaned from this visit will be shared with the Review Committee.The rationale for the self-study pilot visit is to assess whether an added nonaccreditation visit by a team of trained field representatives will accelerate program improvement and innovation, including improvement in those areas where programs already comply with the accreditation standards.For the 10-year visit, all programs will be asked to provide an update of their self-study summary (for any changes in aims or environmental context), along with a succinct “summary of achievement” that details program strengths and improvements the program has already achieved as a result of its self-study conducted 12 to 18 months earlier. No information will be requested or collected for areas that the program has identified as still in need of improvement.The information on the improvements achieved will assist the Review Committee in making an initial assessment of the effectiveness of the self-study process, based on the description of the program’s aims and context, and the improvements and innovations that were realized.A fifth important element of the approach to the self-study and the 10-year accreditation site visit is an ongoing, comprehensive program evaluation of this dimension of the new accreditation system that will gather input from program and institutional leaders; Review Committee chairs, members, and staff; and accreditation field representatives.The structure of the program evaluation and the areas to be evaluated are shown in the figure. Data will be collected in 3 areas that respectively will constitute a needs assessment, an evaluation of the processes for the self-study and the subsequent accreditation site visit, and an outcomes evaluation.The needs evaluation will assess what information and added resources may assist programs in conducting the self-study, including the guidance to assist programs in setting aspirational aims, and exploring their institutional, local, and regional context. A second element of the needs evaluation relates to the learning needs of the Review Committees and ACGME staff regarding evaluating programs’ self-study and the improvement and innovation achieved.The process evaluation will assess whether the new approach to the self-study allows programs to conduct a thorough review of their performance, with a robust, frank discussion of the current state, plans, successes, and areas in need of improvement. For programs in the self-study pilot visit, it will also assess whether the feedback received during the added nonaccreditation visit is thought to be helpful in providing an external perspective and whether this accelerates program improvement, with a particular focus on areas where the program already complied with the accreditation standards.Another area of the process evaluation will explore the perceptions of the benefit of the self-study pilot visit, and perceptions of the burden of an additional visit on programs, sponsoring institutions, and the ACGME. The process evaluation will also seek to determine whether 12 to 18 months between the self-study and the 10-year site visit is sufficient time to address compliance problems, and to begin to realize improvement in areas identified during a program’s self-study. Finally, the process evaluation will seek to identify effective approaches and best practices for how to conduct a program self-study.Among other aspects, the outcomes evaluation will compare improvement outcomes for programs that participate in the self-study pilot visit to a comparison group of programs that will not have the self-study pilot visit but volunteer to share data to facilitate the assessment of the value of this added visit. Deidentified matched data will be collected for both groups to explore whether the added nonaccreditation site visit will accelerate improvement or result in improvements that are more impactful, given the given program’s aims, and the context in which it operates.A key aim of the program evaluation is to allow the ACGME, the Review Committees, and the graduate medical education community to learn from the early self-study process. For all dimensions of the program evaluation, only aggregated, deidentified data will be shared, with a focus on examples of effective program improvement and best practices for planning and executing the self-study, including how programs may use the self-study document as a “living document” that has ongoing usefulness after the self-study is completed.The ACGME plans to disseminate information learned through this approach via its website, in meetings and webinars, and through the Journal of Graduate Medical Education.Through the self-study pilot and the associated program evaluation, the ACGME hopes to promote an intentional design and implementation of accredited residency and fellowship programs, and determine whether programs benefit from an external review and feedback on their self-assessment and its results.Studying the value of an external review and discussion of the self-study with volunteer programs will give the ACGME insight into whether an added nonaccreditation visit should be put in place for all programs, or continue to be offered on a voluntary basis. This decision will need to be made from evidence that this voluntary site visit adds value to the self-assessment and improvement process that is a key component of the new accreditation system.