Answers are grounded in this paper and cite their sources.
Being sufficiently "data literate" (e.g., Tableau, 2023) is a core competency across most professions in the world today. Being data literate may run the gamut from having more advanced computational training to having a more applied understanding of the power of use (and misuse) of data for different applications. The latter is nowadays formalized in the role of a "data champion" (e.g., Clare, 2019) whose primary goal is to advocate for and support the creation of a coherent "data culture" with associated processes for data architecture, engineering, and governance (e.g., Danese & Gatti, 2023). Working with data is obviously an essential part of the discipline of educational measurement as well—in particular when it comes to latent-variable-based methodologies—but, as I will discuss in this commentary, it is increasingly not a uniquely differentiating characteristic. Specifically, in the last 20 years or so, the conceptual boundaries of the field of educational measurement have become increasingly fluid, starting to integrate more deeply with neighboring disciplinary areas within an increasingly interdisciplinary space. These neighboring areas include heavily computational domains such as general statistical methodology (e.g., Rabe-Hesketh & Skrondal, 2004), computational psychometrics (e.g., Hao et al., 2024; von Davier et al., 2021), data science (e.g., AWS, 2023; He & Lin, 2020), multimodal learning analytics (e.g., Ouhaichi et al., 2023), large language models (Naveed et al., 2023), automated scoring (e.g., Yan et al., 2019), and artificial intelligence (e.g., IBM, 2023) on the one hand and heavily foundational domains such as UI/UX design (e.g., Lamprecht, 2023), learning sciences (e.g., Tokuhama-Espinosa, 2019), sociocultural psychology (e.g., Mislevy, 2018), and justice-oriented antiracist assessment (e.g., Randall, 2021) on the other hand. So, what does this interdisciplinary proliferation of concepts, tools, and practices imply about the way we develop, nurture, and identify talent in educational measurement? The authors of the NCME Presidential Task Force report presented in this issue (Ackerman et al., 2024) did a fabulous job at painting a comprehensive and differentiated big picture of the connectedness of these issues with regard to relevant training, professional experiences, and the changing nature of our profession. I genuinely appreciate how they were able to pull various complicated conceptual strands together and paint an accessible picture for a wide readership on these issues. As such, I do not want to offer up a point-by-point professional critique of their work with an unnecessary nit-picky negative spin. Rather, I would like to embrace their report as an opportunity to reflect on experiences and perspectives that I have developed throughout my career across academia, industry, and K–12. Some of these connections will be a bit tighter while others will be a bit looser, given the limitations of space for the commentary. I offer up these additional expositions on issues raised in the main report in order to help those who are entering this interdisciplinary professional space—or those who are changing job trajectories within it - make some additional sense of the complexities of our heavily evolving work. I focus in particular on (1) the role of artificial intelligence (AI) for transforming educational experiences, assessment, and measurement and (2) tightening the connection between professional work in educational measurement writ large in professional occupations, postsecondary education, and K–12 education. Even though this commentary was not produced by ChatGPT—for better and for worse as some might say! —it is conceivable in 2024 that an initial draft could have been. As most of us can attest, recent advances of generative large language models, image, visual art, or storyboard generation, music creation, code production, and other AI-based tools across disciplines have been nothing short of mind-blowing. For example, in my recent work I have used ChatGPT to simulate and visualize data, summarize thematic ideas from open-ended responses in surveys, write draft text for publication materials, help me generate and refine ideas, create graphics for presentations, and identify possibly valuable resources. With appropriate checks for accuracy and appropriateness of course! These advances have already transformed various job markets, with both positive and unfortunate consequences for the lives of individuals (e.g., World Economic Forum, 2023). These conditions necessitate that anyone who wants to be competitive in the job market in the future develop a lifelong growth mindset to facilitate consistent upskilling based on technological advances more generally and AI advances more specifically. This is as true for most disciplines generally as it is for the discipline of educational measurement specifically. The good news is that this demand for constant upskilling creates a powerful opportunity space for innovations in learning experiences and, consequently, various forms of assessment experiences at scale (e.g., Rupp & Lorié, 2023). In this regard, a recent report by the US Office of Educational Technology (Cardona et al., 2023) painted a comprehensive picture of the impact of AI on education. The authors of the report consistently underscored the power of AI for learning, especially formative instructional guidance, and assessment as and for learning. For example, there will likely be an increasing hunger by various stakeholders for flexible, adaptable, and (at least partially) automated learning and assessment systems that are coherent and well-integrated into curricular experiences. Beyond more advanced integrated systems, there will also be an ever-increasing desire for more authentic project-based learning activities and associated performance assessment events that can only be made possible by the thoughtful application of AI-supported digital tools due to the advances in the disciplinary areas from which the problems are borrowed. These advances will of course not make human educators obsolete, but, rather, require them to adapt and shift their roles and responsibilities in the classroom and when working with school leaders, parents, and community members. In addition to educators needing to adjust the way they approach education, the demand for stakeholder engagement during the design, implementation, and evaluation of these systems will continue to increase more generally. We can already see that broader societal awareness of biases, risks, and inequitable decision-making is rising, and more and more traditionally marginalized groups affected by these tools and the associated processes consequently want their voices to be heard and acted upon in the hopes of democratizing educational processes and outcomes (e.g., Mascareñas & Tran, 2023). More narrowly, advances in AI require educational measurement specialists to re-examine the role of "humans in the loop" during the design, implementation, evaluation, and refinement of such technologies. Ironically, the "humans" referred to in these contexts are typically—either explicitly or implicitly—highly trained, specialized experts, which means that their expertise will likely have been vetted by people in power positions. Yet, even such expert involvement is not a global panacea as even experts can be fallible to ignoring biases while chasing technical innovations (e.g., Turner et al., 2023). Arguably, for certain kinds of judgment processes of AI-generated content (e.g., appropriateness, sociocultural biases) the humans in question should certainly include representatives from broader stakeholder groups so as to not perpetuate existing blindspots. Advances in AI and the applications that this advancement supports will probably lead to an(other) identity crisis for the discipline of educational measurement. Our field has historically been obsessed with norm-referenced interpretations and associated scaling technologies in application contexts with heavy design and implementation constraints. While there is clearly a role for place for these kinds of applications—consider, for instance, national and international educational surveys—there are also clearly design-based limits for what these assessments can tell us. For example, NAEP and PISA are predominantly designed to provide brief, coarse-grained monitoring snapshots of achievement and drivers for achievement for groups of students based on representative samples. Despite innovations in the way complex competencies are operationalized in these contexts (see, e.g., competencies such as collaborative problem-solving) and optimism for deeper transformations in the future (e.g., Foster & Piacentini, 2023), there is still a large conceptual gap between the nature of these tasks and the richness of the evidence that is being collected about learners due to the complex constraints of the survey implementation context. One notable opportunity for growth in the discipline of educational measurement is in supporting the principled design, deployment, and evaluation of interactive, integrated learning and assessment environments that allow for individual and collaborative problem-solving at scale. This work requires embedded data analytics applied to logfiles/trace files that can be appropriately fine-tuned and then leveraged for feedback to support competency development over time. But it also requires a current understanding of what the learning and applied cognitive sciences as well as experiences from the field tell us about how learning happens for learners with different personalities, needs, and backgrounds (e.g., Brown, 2022). The interdisciplinary toolkit that is required to do this work rigorously and meaningfully in specific educational contexts can be generative and insightful. That is, it can in turn be applied to nationally and internationally scaled assessment endeavors for a broad variety of organizational decision-making purposes. At a US-national level, this is already happening with organizations such as Inq-ITS (Apprendis, 2023) or Curriculum Associates (CAINC, 2023) in the K–12 space. At the international level, this is already happening with organizations such as Duolingo (Burstein et al., 2022), Khan Academy (diCerbo, 2024), Amazon (Amazon, 2023), or Tesla (Tesla, 2023) in the adult learning space. All of this work undoubtedly requires some large-scale data-processing methods from somewhere in the intersection of general statistics, data science, and psychometrics, but I doubt that, in the long term, latent-variable-based scaling methodologies such as item response theory (e.g., van der Linden, 2021) or structural equation modeling (e.g., Kline, 2023) will be the exclusive or even dominant tool for data aggregation. Nor will the most innovative analytic methods most likely have been developed exclusively by specialists in educational measurement. For example, consider (dynamic) Bayesian networks, which is one powerful modeling framework for these kinds of applications. While they have been brought closer to the attention of educational assessment specialists through publications such as Almond et al. (2015), modern web-based resources quickly underscore how applications in education are just one of many examples of these (e.g., Bayes Server, 2023). Therefore, one may ask whether the value proposition for a graduate degree in educational measurement can remain strong enough for the job market of the future as it will have to compete with the value proposition offered by similar kinds of technical and computational degrees from neighboring disciplines. To help postsecondary students understand these value propositions more deeply so that they can continually make informed decisions about their career pathways, a closer connection of professional workspaces and postsecondary education is necessary. Learners of all ages, starting as early as upper middle and/or high school, are asked to make a variety of "bets" on what an appropriate set of skills to develop might be in order to have reasonable chances of entering different career pathways. This is reflected in the projects they choose in courses, the high school courses that they select, and the reflections they are asked to make about their plans for the future, to name but a few situations. That is a tough challenge for anyone at any given point in time, let alone repeatedly over time and with distal time horizons. Understanding how different professional areas are (currently) structured, what work in these areas (currently) looks like, at least prototypically, and what knowledge, skills, and competencies one (currently) requires is relatively hard to parse for most of us, at least in my experience. Moreover, not only are professional disciplines themselves transforming at a rapid pace due to advances in technology and AI, as discussed in the previous section, each discipline is also rather diverse within its own conceptual boundaries at any given point in time. It is helpful in this regard to recognize prototypical differences across the resulting professional subspaces that affect the nature of the work that is being conducted. The focal report by the NCME Presidential Task Force describes essential interdependencies between required competencies and different jobs in the section on Educational Measurement Careers. As the authors aptly noted, entry-level skills that one has developed through badges, certificates, or a master's degree are indeed just a starting point for a career trajectory that can take on very different shapes depending on a variety of factors. There are typically opportunities for utilizing a particular skill set in many different jobs, but the relative importance, time spent, and practical impact of the work will differ notably in these jobs. For example, while simulation studies are a common tool in our field, they are most frequently employed as part of PhD pursuits in heavily methodological programs, sometimes employed as part of analytic development processes in ed tech contexts, but relatively rarely employed as part of the work by district or state assessment experts. Such characterizations should obviously be taken with some grain of salt as the experiences of individuals will always differ in important ways. Beyond such interindividual differences, there are also additional structural distinctions within disciplinary subspaces that matter. That is, there is also no such thing as "the university experience" or "the industry experience" or "the K–12 experience." Rather, there are important differences across the organizations within these broader subspaces and typically even within certain departments, divisions, or units within these. For example, studying or working on educational measurement issues in a larger, research-centric university program is very different from studying or working on these issues in a smaller, teaching-centric university program, even though both research and teaching matter to some degree in each of these programs. Similarly, working on these issues in a small ed-tech start-up is very different from working on them in a larger, established organization that may function effectively like different independent suborganizations where this work is central in some of them and irrelevant in others. As a final example, working as an assessment or accountability specialist in a small or midsize district is very different than working in this role at a state department or independent consulting organization, partly due to the availability and structure of data and the nature of the problems that are being solved (e.g., local school improvement vs. state-wide system monitoring and resource allocation). The degree to which these differences are important and influential for one's career path at any given point in time depends on several factors, some within one's control and some beyond it. For example, one has essentially no control over the core demands of a job or how the characteristics of a given employment sector change over time, which affects internal reorganization or may lead to layoffs. However, one has much more influence over how one reacts to induced shifts as one can try to adapt, grow, and change to meet the emerging demands of a sector. I am definitely not saying that this is equally possible for everyone under all conditions, but I am hopeful that, with an increased modularization of learning experiences and associated recognition of skills and more freely available or, at the very least, low-cost educational opportunities, the playing field will slowly become more level for professionals with a variety of backgrounds and experiences. In this regard, NCME has started offering initial digital training modules under the umbrella of their ITEMS series (NCME, 2023), which offers a fruitful starting place for developing more formal courses, badges, and certificates in collaboration with partner organizations. One of the peculiar aspects of educational measurement as a discipline that I have observed over the years is a notable disconnect between the high degree of technical specialization in advanced modeling commonly afforded by a "prototypical" PhD education with a methodological focus and the much higher degree of reliance on a wide array of applied, interdisciplinary skill sets in real-life jobs. This perception was echoed, to some degree, by a recent curriculum review of educational measurement programs (Randall et al., 2021). Specifically, the authors pointed out that there is a solid coverage of core technical content in most programs in line with what experts believe is important but a notable lack of professional development opportunities and graduate courses in important areas that are relevant to decision-making with empirical assessment information such as educational policy. Some programs build such bridges rather effectively while others are much more narrowly focused and might simply offer elective courses in other departments that are not truly integrated with the learning in the core program. This affords very different opportunities for students across programs and, thus, different entry points to the professional space. Arguably, then, to better prepare students for the future, we need more opportunities for them to develop flexible but rigorous habits of mind along with experience with diverse digital tools through rich, authentic, contextualized decision-making problems. For example, I could see a strong value for expanded educational measurement programs at the master's level in the future that offer opportunities for learners to develop foundational computational and interdisciplinary competencies, enriched through a variety of rich problem-solving contexts, ideally structured in such a way that the outcomes of their work actually matter to real-life practice (e.g., helping an organization improve their reporting practices or shape educational policy). Of course, the best academic programs already offer such opportunities through the connections that faculty have with businesses, government organizations, and other institutions and more of that is needed. In such a world there would remain a role for PhD programs of course but I could see those more narrowly targeted toward the kinds of students who truly want to advance the quantitative methodological (or at least mixed-methods) machinery per se and enter a life of teaching and graduate student mentoring in these programs. Yet, I would argue that there is a risk in over-privileging (current) PhD training in educational measurement for certain professional positions when, in fact, a well-targeted master's degree coupled with diverse job experiences and strong qualifications in so-called "skills for the future" (a.k.a. "durable skills," "transportable skills," or "complex skills," among others; e.g., Prichard Committee, 2022) would do equally well, if not better. This mindset change in hiring practices would also reduce the undue financial burden that is placed on students by extensive, drawn-out graduate programs and open up career opportunities for a wider range of job candidates. In other words, if we do not change the ways we develop and nurture talent as well as look for talent when we need it, we will not be able to change our talent pool and leadership structures. Perhaps ironically, this argument mirrors debates on the importance of developing skills for the future in K–12 education. For example, in 2023, 17 states had officially adopted a portrait of a graduate (PoG) that can act as an inspirational north star and guardrail for districts across these states as they are designing their own PoGs (Stanford, 2023); this number has likely already increased. A PoG is a general description of the core competencies or profiles that students are expected to develop throughout their K–12 career, allowing them to apply foundational literacy skills in core domains to a variety of complex tasks by themselves and in collaboration with others. Consequently, the adoption of PoGs is often coupled with learner-centered, competency-based education (CBE) models (e.g., Erwin & Silva-Padrón, 2022; Evans et al., 2020) and an associated renewed interest in modernized versions of project-based learning, portfolio-based demonstration of learning, and exhibitions or defenses of learning at essential transition grades and beyond (e.g., KAEC, 2023). These programs offer up opportunities for connecting K–12 students to learning and assessment issues that might open their eyes to how they can participate in, and importantly shape, the future of this work. With regards to the last point, learners who are discontent with the education system they are experiencing should be encouraged to co-design creative education and assessment solutions and tackle their points of dissatisfaction systematically in collaboration with education leaders. Increasing awareness about the relevance of educational assessment and measurement issues can include simple tactics such as having people from key local, state, or national organizations visit local schools and present real problems of practice along with the realistic constraints—either virtually or in person. These guests can also describe what a typical day, week, month, and year in their roles look like and how they are using the skills they have been learning in both predictable and unique or unexpected ways. Teachers can also leverage the availability of existing public data dashboards and data sets on educational issues to explore issues around data quality and governance, statistical analysis, and educational policy. Districts can develop more interdisciplinary courses that bridge educational and assessment issues by, for example, having students develop assessment processes and validation work around the projects they are completing. The best teachers are already doing this of course. In addition, CBE models make room for alternative models of formally recognizing the development and acquisition of these competencies in the form of badges and micocertificates (e.g., Allen et al., 2022; XQ, 2023). This customization and modularization of educational experiences is already commonplace in various spaces such as YouTube, Khan Academy, Coursera, or EdX, to name but a few. In the dedicated postsecondary space, companies and universities have similarly begun to compete for the financial resources of students by offering certificates and entire graduate degrees online (e.g., Credly, 2023; Western Governors University, 2023). In this regard, the focal report by the NCME Presidential Task Force in this issue lays out a powerful framework for developing such partnerships and resources. Educational measurement competencies will remain important in the future as AI-fueled routines are increasingly used to generate a wider variety of stimuli, complex sample responses, and response evaluation routines. However, an agile adaptation of best practices, professional standards, and general guidelines will be needed to manage the increasing complexities of detecting, understanding, and addressing undesirable side effects. Our collective historical status quo of what assessments looked like in the past, how they were evaluated, and how they were being (mis)used in various context cannot serve as the sole, or even primary, guardrail for this work. In this sense, the joint initiative by AERA, APA, and NCME to update the standards of the profession—and, I would expect, the processes and timelines by which these are reviewed and updated over time—is timely (AERA, APA, & NCME, 2023). Moreover, there is certainly no way around it: Professionals who are interested in more advanced data-based reasoning competencies for solving educational problems of practice that matter at scale will have to continue to adapt to the changing nature of the interdisciplinary space that their work is situated in. Competencies in educational measurement as traditionally conceived will be part of this skill set but I imagine that the field itself will become less attractive as a stand-alone field with a unique value proposition. This seems to be signaled already by the challenges around attracting new students to the discipline that I have heard various professionals articulate within the NCME Educators of Educational Measurement SIGIMIE and more broadly in casual conversations at recent conferences or other gatherings. The good news is that the changes surrounding our field offer up rich playgrounds for advancement and growth; we just have to find productive and generative ways to leverage them.
No takes yet. Share an insight, caveat, or question.
André Rupp (2024) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: