Preliminary evaluation of an AI-powered chatbot for academic integrity support in higher education

This study reports a preliminary internal evaluation of the A-Integrity Buddy chatbot, an AI-powered tool designed to provide policy-aligned academic-integrity guidance within one Australian university. Using a scenario-based quantitative design, 84 author-developed scenarios grouped into author-assigned Easy (n = 30), Medium (n = 27), and Difficult (n = 27) categories were submitted to the chatbot. The first generated response to each scenario was retained and rated separately, without consultation, by three institution-based experts on a five-point scale for policy alignment and response adequacy. The same expert panel had contributed to scenario development; accordingly, the study represents an internal formative benchmark rather than independent validation. Across 84 scenarios, the overall mean scenario rating was 4.56 out of 5 (95% bootstrap interval [4.45, 4.65]). Mean scenario ratings were 4.71 (95% bootstrap interval [4.60, 4.81]) for Easy, 4.43 (95% bootstrap interval [4.16, 4.68]) for Medium, and 4.52 (95% bootstrap interval [4.40, 4.63]) for Difficult scenarios. Overall ordinally weighted Gwet’s AC2 was 0.911 (95% CI [0.886, 0.935]), although exact three-rater agreement occurred for 39.3% of scenarios. The high weighted coefficient therefore reflected agreement or near-agreement, predominantly within the upper rating categories. These findings indicate generally favourable internal expert appraisal of the fixed chatbot outputs within the evaluated institutional policy context. Interpretation is constrained by the small non-independent rater panel, the single-institution setting, and the use of one generated response per scenario. Further independent and end-user evaluation is required before claims can be made about reliability, educational impact, or generalisability. Not applicable.