An empirical mixed-methods evaluation of AI-based chatbots for teacher professional development in Austrian higher education

Purpose The continuous professionalization of teachers is crucial for sustaining high-quality education. However, traditional professional development (PD) programs often neglect individual needs, specific subject-area demands and distinct career stages, leading to limited relevance and uptake. This study addresses that gap by deploying an AI-based chatbot to provide context-sensitive, personalized PD recommendations at scale. Grounded in technological pedagogical content knowledge (TPACK)and self-determination theory, the research aims to evaluate how tailored chatbot interactions can enhance teachers' motivation, autonomy and technological proficiency while meeting pedagogical and content-specific requirements. Design/methodology/approach Using a convergent parallel mixed-methods design, this study analyzed 2,030 valid chatbot interactions from 1,125 teachers in Austria's Burgenland region. Data collection incorporated the information systems success model (ISSM), the technology acceptance model (TAM) and TPACK as guiding frameworks. Quantitative metrics included fallback rates, implicit intent interpretation, sentiment analysis and confidence scores, whereas qualitative feedback examined perceived relevance. Descriptive and inferential statistics, alongside content analyses, were used to assess the chatbot's performance. This design enabled a comprehensive evaluation of both measurable indicators and user perspectives regarding chatbot-enabled PD recommendations. Findings Results demonstrated a moderate fallback rate of 14.4%, significantly below established benchmarks and an overall positive user sentiment (85%). Quantitative analyses indicated that teachers submitting highly specific queries reported greater satisfaction, while logistic regression revealed that targeted pedagogical keywords significantly increased the likelihood of positive feedback. Qualitative insights underscored the importance of both detailed query formulations and domain-specific terminology. Collectively, these findings highlight robust chatbot performance and emphasize the critical role of contextualized, technology-oriented PD solutions for meeting teachers' individualized professional needs. Research limitations/implications Due to the relatively brief observation period and the self-selecting nature of participating teachers, these findings may not be generalizable across broader educational settings. The sample, drawn from a single Austrian region, may limit external validity. Future research should incorporate larger, more diverse populations, extend the timeframe to measure long-term outcomes and collect additional demographic data to assess subgroup variations. Longitudinal investigations into the sustained impact of chatbot-based recommendations on teaching practice can further elucidate the role of AI-driven PD in different educational contexts. Practical implications Institutional stakeholders can optimize AI-based PD tools by encouraging teachers to submit more detailed queries and employ targeted pedagogical terminology. Additionally, systematic refinements, such as updating domain-specific vocabularies and improving natural language processing algorithms, can reduce fallback rates and enhance user satisfaction. Training programs aimed at familiarizing educators with chatbot functionalities and best practices can further increase engagement. By aligning professional development offerings with teachers' immediate needs and contexts, these strategies can strengthen the relevance, accessibility and overall impact of AI-driven PD initiatives. Social implications By providing accessible, context-sensitive PD resources, AI-driven chatbots may help democratize professional learning for teachers across diverse settings, including those with limited institutional support. This can contribute to narrowing digital skill gaps, especially in remote or underserved schools, thereby promoting educational equity. Enhanced teacher engagement in technology-enhanced PD aligns with broader objectives of fostering lifelong learning cultures and continuously improving educational quality. As teachers refine their digital competencies, they may experience greater autonomy and motivation, fostering a ripple effect on student outcomes and broader societal advancement. Originality/value This research uniquely synthesizes the ISSM, TAM and TPACK to evaluate chatbot-supported teacher PD, offering a multi-faceted assessment of both user experience and educational relevance. By emphasizing the significance of query specificity and targeted pedagogical language, the study advances understandings of how AI-driven tools can address individualized teacher needs in diverse contexts. The findings deliver practical guidance for refining chatbot technologies and theoretical insights into the interplay of technology acceptance, pedagogical content knowledge and AI-based support systems, thereby contributing to ongoing discourse on data-informed professional development.