Rubrics on the pitch: Exploring the reliability of soccer scout's assessments

Soccer clubs use rubrics to guide scouts and coaches toward uniform rating standards, yet empirical evidence on their reliability remains limited. Therefore, this exploratory field study examined the inter-rater reliability of scouts’ assessments during an 11v11 live match using a rubric. We focused on current and potential performance, information combination methods, and scouts’ confidence and perceived difficulty. Sixteen male scouts ( Mage = 55 years, Mexperience = 6 years) each evaluated three to four under-13 players (yielding 23 unique players in total), rating seven rubric performance indicators on both current and potential performance: motoric/physical, technical execution, attacking on the ball, attacking positioning, defensive duels, defending positioning, and transitioning. Overall current and potential performance scores were established in two different ways: based on the scouts’ judgments (i.e., clinical combination), and by calculating the mean of their indicator ratings (i.e., actuarial combination). Across both these combination methods, inter-rater reliability was moderate for overall current performance (ICC = .61 clinical, ICC = .61 actuarial), but poor for overall potential performance (ICC = .37 clinical, ICC = .42 actuarial). Clinical versus actuarial combination made little difference in reliability. Moreover, scouts reported similar confidence assessing potential but greater difficulty compared to current performance. The findings suggest that evaluations of potential are especially prone to unreliability and highlight the need to further examine reliability in soccer scouting.