What does this research mean for the field?

Current evidence supports the use of large language models for carefully bounded formative and human-supervised second-language assessment, but does not justify their use for autonomous high-stakes scoring due to underdeveloped construct representation, fairness, and generalizability. Novelty: ClaimNovelty.SYNTHESIS. Consensus alignment: ConsensusAlignment.NEUTRAL.

What question did this study set out to answer?

To evaluate when large language model scoring and feedback are psychometrically and educationally defensible in second-language assessment using an argument-based validity framework.

September 14, 2026Frontiers in Research Metrics and AnalyticsOpen Access

Conditional validity in LLM-mediated L2 assessment: an argument-based systematic review and meta-analysis

View Full Paper

Ask AI

Bookmark

Authors

LALatifah Hamdan Alghamdi TATalal Musaed Alghizzi

Discussion

Member takes

Overview

Systematic review and meta-analysis reveals moderate validity for LLM scoring in second-language assessment, indicating that autonomous high-stakes testing remains unsupported.

Key Points

To evaluate when large language model scoring and feedback are psychometrically and educationally defensible in second-language assessment using an argument-based validity framework.
Followed PRISMA 2020 guidelines to systematically search nine databases, identifying 52 empirical studies published between January 2022 and December 2025.
Conducted meta-analyses using REML random-effects models to pool Pearson correlation coefficients for scoring correspondence and Hedges' g for formative feedback outcomes.
Human-LLM score correspondence showed moderate agreement across three verified studies (r = 0.66, 95% CI [0.53, 0.75], I² = 70.8%), with the strongest validity observed under rubric-guided and human-supervised conditions.
Formative feedback mediated by language models yielded a small-to-moderate positive effect on revision quality and short-term writing performance across 14 studies (Hedges' g = 0.42, 95% CI [0.27, 0.57], I² = 69.8%).

Cite This Study

Alghamdi et al. (2026) studied this question.

synapsesocial.com/papers/6aa7b2e10926e14a848b1696 https://doi.org/10.3389/frma.2026.1908592

View Full Paper

Ask AI

Bookmark

September 14, 2026Frontiers in Research Metrics and AnalyticsOpen Access

Conditional validity in LLM-mediated L2 assessment: an argument-based systematic review and meta-analysis

View Full Paper

Ask AI

Bookmark

Authors

LALatifah Hamdan Alghamdi TATalal Musaed Alghizzi

Discussion

Member takes

Overview

Systematic review and meta-analysis reveals moderate validity for LLM scoring in second-language assessment, indicating that autonomous high-stakes testing remains unsupported.

Key Points

To evaluate when large language model scoring and feedback are psychometrically and educationally defensible in second-language assessment using an argument-based validity framework.
Followed PRISMA 2020 guidelines to systematically search nine databases, identifying 52 empirical studies published between January 2022 and December 2025.
Conducted meta-analyses using REML random-effects models to pool Pearson correlation coefficients for scoring correspondence and Hedges' g for formative feedback outcomes.
Human-LLM score correspondence showed moderate agreement across three verified studies (r = 0.66, 95% CI [0.53, 0.75], I² = 70.8%), with the strongest validity observed under rubric-guided and human-supervised conditions.
Formative feedback mediated by language models yielded a small-to-moderate positive effect on revision quality and short-term writing performance across 14 studies (Hedges' g = 0.42, 95% CI [0.27, 0.57], I² = 69.8%).

Cite This Study

Alghamdi et al. (2026) studied this question.

synapsesocial.com/papers/6aa7b2e10926e14a848b1696 https://doi.org/10.3389/frma.2026.1908592

Also Consider

Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context:

View Full Paper

Ask AI

Bookmark