Modeling the joint impact of human and AI metacognitive sensitivity on human–AI collaboration

Successful human–AI collaboration requires accurate uncertainty communication. While prior research emphasizes calibration, we demonstrate that metacognitive sensitivity – the ability of confidence scores to discriminate between correct and wrong decisions – is equally important. We derive the Bayes-optimal decision rule for combining human and AI predictions and use signal detection theory to analytically express combined accuracy based on human and AI metacognitive sensitivities. We prove that above-chance metacognitive sensitivity in either agent guarantees complementarity, where joint accuracy exceeds both individual accuracies. Monte Carlo simulations and empirical validation on human–AI image classification demonstrate that our analytic solution remains robust to non-Gaussian confidence distributions. These findings establish metacognitive sensitivity as an important determinant of human–AI collaboration.