What question did this study set out to answer?

To benchmark the capability of generative language models to act as automated teachers generating conversational responses in teacher-student dialogues.

January 1, 2023Open Access

The BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues

View Full Paper

Ask AI

Bookmark

Authors

ATAnaïs TackImec the Netherlands EKEkaterina KochmarMohamed bin Zayed University of Artificial Intelligence ZYZheng YuanUniversity of Sheffield

Discussion

Member takes

Implication

Shared-task evaluation demonstrates large language models can generate effective teacher responses in student dialogues, indicating a need for metrics tailored to pedagogical quality.

Key Points

To benchmark the capability of generative language models to act as automated teachers generating conversational responses in teacher-student dialogues.
Benchmarked submissions across eight participating teams evaluating models such as GPT-4, GPT-3.5, LLaMA, Bloom, and Flan-T5 on teacher response generation.
Evaluated generated responses automatically using BERTScore and DialogRPT metrics, followed by human expert pedagogical assessment of the top three systems.
The top-ranking system (NAISTeacher) utilized GPT-3.5 prompt ensembling paired with DialogRPT response reranking to achieve the highest scores in both automated and manual pedagogical evaluations.
Existing automated metrics demonstrated limitations in capturing pedagogical nuances, revealing a critical need for domain-specific evaluation methods in educational AI dialogue.

Cite This Study

Tack et al. (2023) studied this question.

synapsesocial.com/papers/6a106dd75725bbd5cc60ff22 https://doi.org/10.18653/v1/2023.bea-1.64

View Full Paper

Ask AI

Bookmark

Implication

Shared-task evaluation demonstrates large language models can generate effective teacher responses in student dialogues, indicating a need for metrics tailored to pedagogical quality.

The BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues

Discussion

Member takes

Implication

Key Points

Cite This Study

The BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues

Discussion

Member takes

Implication

Key Points

Cite This Study

Also Consider

Also Consider