Adjusted Positive Agreement: A Chance-Corrected Agreement Measure for Unconstrained Pile Sorts

Unconstrained pile sorting, also known as free pile sorting or card sorting, is an important method for studying how people organize items within a domain. Yet commonly used measures for estimating respondent agreement struggle with the lumper–splitter problem. As a result, apparent similarity between respondents may reflect propensities to lump or split rather than shared conceptual agreement. This article introduces the adjusted positive agreement (APA), a pair-counting coefficient designed to address this problem. APA compares the item pairs that two respondents place together, adjusts this overlap for the total number of co-placed pairs produced by each respondent, and scales the result relative to the smaller of the two respondents’ co-placement totals. Across two simulation conditions, APA preserved agreement with less influence from lumping and splitting than Pearson correlation and the adjusted Rand index. The findings support APA as a basis for analyzing how cultural models are distributed across respondents.