There is growing interest in identifying and reducing noise, the unwanted variability in subjective evaluations, in personnel selection. Although a framework for noise analysis exists, its limited use may stem from a lack of tools to identify and evaluate the magnitude of noise. This article aims to present both technical and non-technical options for noise analysis and evaluation, and to provide a general threshold for acceptable noise levels along with qualitative descriptors. It also bridges the concepts of noise and interrater agreement, clarifying how different measures of agreement can inform noise analysis. First, several quantitative tools for noise analysis are demonstrated, including partitioning noise into level and pattern components, quantifying noise as a percentage, and assessing complex interrater reliability metrics such as the Intraclass Correlation Coefficient (ICC) and Gwet’s AC. Examples and code are provided in Excel, R, and Stata. Second, qualitative descriptions and a threshold for acceptable and unacceptable noise levels are benchmarked using data from a subject matter expert survey
Nordmo et al. (2025) studied this question.