Not Interchangeable: Pilot Self-Reports and Instructor Competency Ratings as Cognitive-State Labels in Real Flight Operations

Objective To quantify the correspondence between pilots’ post-task workload and situation-awareness ratings and instructor competency ratings during real flight. Background Adaptive systems, neuroergonomic classifiers, and competency-based training all need cognitive-state labels. Self-reports capture felt demand and awareness; instructor evaluations capture behaviorally inferred competence. Method Forty pilots completed five Cessna 172 tasks during dedicated research flights. After each task, pilots gave single-item 0–10 ratings of workload and situation awareness. One flight instructor at each of the two participating organizations rated Situation Awareness and Workload Management on 1–4 International Civil Aviation Organization-aligned scales, and rated task execution with behavior-specific items, before hearing the self-ratings. Analyses used Spearman correlations with participant-clustered bootstrap intervals, within-person decomposition, and mixed-model and stratified sensitivity analyses. Results Pooled cross-source correspondence was weak for situation awareness ( ρ = . 08 ) and workload ( ρ = − . 14 ; . 14 aligned). The two instructor competencies were strongly intercorrelated ( ρ = . 61 ) and strongly associated with same-instructor task-execution ratings ( ρ = . 69 and . 72 ). A small within-person workload signal ( r = − . 19 ) attenuated after double centering. Point estimates were similar in both strata and after excluding pilots aged 60 or older. Conclusion The two sources provided distinct, non-substitutable labels. Their strong link to same-rater task-execution scores is consistent with execution-referenced judgment, though shared-rater variance cannot be ruled out; self-reports captured pilots’ reported experience. Application Cognitive-state monitoring and training assessment should use multi-source labels and should not treat observation-based competency ratings as direct measures of internal workload or situation awareness.