Can Occupational Safety Practitioners Trust AI-Ranked Risk Factors? A Synthetic Study of Predictive Performance and Explanation Fidelity

Artificial intelligence tools are increasingly used in occupational safety to rank factors associated with incidents and to support decisions on training, supervision, inspection, and preventive resource allocation. We evaluated whether predictive performance and explanation fidelity remain aligned when events are rare, and predictors are correlated. A fully synthetic occupational system was generated with Safety Data Sheet competence, six Bloom’s Taxonomy dimensions, behavioral and organizational variables, and a simulated incident outcome. Random forests and ridge models were assessed for nested cross-validation discrimination and calibration; random forests were additionally evaluated using ablation, held-out marginal and conditional permutation feature importance, and matched oracle importance derived from the known data-generating mechanism. A 29-condition Monte Carlo experiment (58,000 random-forest fits) examined sample size, prevalence, effect magnitude, and predictor architecture. Under the baseline condition, mean AUC was 0.689, whereas oracle–model rank agreement was weak (Spearman ρ=0.048) and oracle top-1 recovery was 20.7%. A high-signal implementation check produced ρ=0.964, with 83.4% top-1 and 98.5% top-2 recovery, while a more intensive rare-event pipeline left top-1 recovery essentially unchanged at 20.3%. Conditional permutation importance, benchmarked against a matched conditional oracle, reduced spurious first-ranking of true-null controls (26.9% to 22.9%, p<0.001) but did not improve recovery of the oracle top variable (20.4% versus 20.7%). AI-derived factor rankings should therefore be validated separately from predictive performance before they are used to set preventive priorities within conventional occupational risk-assessment processes. The evidence presented here concerns random forests with marginal and conditional permutation importance under fully synthetic rare-event conditions, and does not generalize to AI explanation methods as a class.