What PISA Measures and What It Misses: A Two-Stage LLM-Based Alignment of IT Workforce Skills with Educational Proficiency

Aligning information technology (IT) workforce demands with educational assessments is essential for bridging skills gaps; yet, no prior corpus maps IT task reasoning to Programme for International Student Assessment (PISA) proficiency levels. This paper introduces a large language model (LLM)-powered framework aligning IT competencies with PISA 2022 and the OECD (Organisation for Economic Co-operation and Development) Learning Compass 2030, drawing on O*NET v30.2 (Occupational Information Network), ESCO (European Skills, Competences, Qualifications, and Occupations) v1.2.1, PISA descriptors and OECD definitions. The framework operates in two stages: Stage 1 aligns 562 IT task statements with minimum PISA 2022 proficiency levels via LLM annotation and cross-model validation; and Stage 2 extends this mapping to the OECD Learning Compass 2030 through the semantic clustering of task embeddings and a bidirectional gap analysis of 95 ESCO transversal skills. Using Gemini 2.5 Flash, 562 tasks are annotated with minimum PISA levels across Mathematical, Reading, and Science literacy (first stage). Annotation reliability is assessed through a five-model cross-validation against a blind human domain expert (treated as a reference benchmark, not a gold standard) on a stratified 100-task sample (17.8% of the corpus), with agreement ranging from fair (Gemini 2.5 Flash, κ = 0.29) to moderate (Claude Haiku 4.5, κ = 0.50; LLaMA 3.3 70B, κ = 0.44). A bias-correction sensitivity analysis confirms that distributional findings remain stable after accounting for the primary annotator’s systematic overestimation, and OLS-calibrated alignment against O*NET ability ratings provides directional plausibility support. Validated tasks are embedded and clustered into 25 technical profiles via K-Means, each classified against OECD dimensions. The framework is extended to 95 ESCO transversal skills in 24 clusters. Bidirectional analysis reveals that, while every PISA proficiency level is engaged by at least one transversal cluster, 33% of these clusters, covering creative, ethical, social–emotional, and dispositional competencies, fall entirely outside PISA’s cognitive scope. This boundary mapping identifies where the PISA-based alignment is valid and where complementary tools are required for a full readiness assessment.