Bias meets Bayes: Bayesian aggregation for peer and self-ranking

Rank aggregation, which aims to combine multiple ordinal rankings into a single, reliable ranking, is widely studied in diverse fields. In academic and professional settings, peer- and self-evaluations play a pivotal role and offer insights into both individual performance and the competence of rankers. However, subjective biases can lead to unreliable or unfair outcomes, especially in a small sample setting. Despite the broad applications of rank aggregation, existing methods often fail to address nuanced biases and the special structure of such data. This paper introduces BayeSPeer, a novel Bayesian hierarchical model designed to jointly estimate latent performance scores and self-assessment biases in peer- and self-ranking data. Through comprehensive simulation studies, BayeSPeer demonstrates superior performance in rank recovery and top-k identification under various levels of sample size, ranking quality, and bias magnitudes. Real-world applications in classroom peer evaluations also highlight the model’s interpretability and practical utility. Our findings showcase the advantages of incorporating structured bias correction rather than omitting self-evaluation, offering a statistically principled and transparent approach to bias-aware ranking in educational and other evaluation contexts.