Development and validation of the AI literacy questionnaire for health professions students using Rasch analysis

Artificial intelligence (AI) is reshaping healthcare and creating new competency requirements for health professionals. However, assessment instruments tailored to this population remain limited, and item-level measurement evidence is scarce. This study aimed to develop and validate an AI literacy scale (AILS) for health professions students using an integrated theoretical framework. The item pool was informed by UNESCO’s AI Competency Framework for Students, the Technological Pedagogical Content Knowledge model, the Kirkpatrick evaluation model, a literature review, and qualitative interviews. Items were refined through Delphi method, content validation, and pilot testing. Data were collected from 473 students. The sample was divided into two subsamples for exploratory factor analysis ( n = 241) and confirmatory factor analysis with Rasch rating scale analysis ( n = 232). The final 33-item AILS comprised four dimensions: Technical Understanding, Ethical Awareness, Practical Ability, and Social Impact. The four-factor solution explained 63.676% of the total variance. The total scale demonstrated high internal consistency (Cronbach’s α = 0.979), and CFA showed acceptable model fit (χ²/df = 2.175, IFI = 0.917, TLI = 0.908, CFI = 0.916, and RMSEA = 0.071). In the Rasch analysis, person and item reliabilities were 0.96 and 0.95, respectively. The person–item map showed that the mean person measure was 1.66 logits above the mean item difficulty of 0.00 logits, indicating limited upper-end targeting. This mismatch may reflect insufficiently difficult items, sample characteristics, previous AI experience, limited curricular emphasis on advanced competencies, or a combination of these factors. The AILS demonstrated evidence of validity and reliability for assessing AI literacy among health professions students. The combined use of factor analysis and Rasch measurement illustrates the potential value of Rasch analysis for evaluating item functioning and person–item targeting. The current scale appears better suited to group-level assessment of foundational and intermediate AI literacy than to distinguishing advanced competence. Further research should develop more challenging items and validate the AILS in independent, multicentre samples. Application in other professional education domains would require discipline-specific adaptation and validation.