Key points are not available for this paper at this time.
Generative AI (GenAI) tools are increasingly used by students in higher education, including in technically demanding engineering courses. However, fluent AI-generated responses may still contain incorrect or incomplete information, creating a risk that students overestimate their reliability. This exploratory study investigates the relationship between students’ perceived usefulness of GenAI and an instructor-benchmarked reference evaluation of model outputs in two digital systems design courses. The study involved voluntary survey responses from 32 students in an undergraduate course at MIUN and 20 students in a graduate-level course at UNISA. Student perception data were combined with teacher-side benchmarking of selected GenAI models on tasks categorized by cognitive depth. Findings indicate that prior GenAI familiarity was associated with interaction frequency and average perceived usefulness, whereas self-assessed subject knowledge showed limited association. A perception–performance gap emerged, with students often rating GenAI outputs as useful even when the instructor-side evaluation identified limitations in correctness or required substantial human scaffolding. The proposed framework should be interpreted as an exploratory guideline for studying and guiding GenAI use in engineering education, rather than as a definitive benchmark of model performance.
Shallari et al. (Wed,) studied this question.