Getting More From Your IOA Data: Alternative Measures to Total, Occurrence, and Non‐Occurrence Agreement

Behavior analysts often need to rely on interobserver agreement (IOA) as a substitute for direct measures of the accuracy of collected trial‐by‐trial data. In these situations, behavior analysts often report the sum of instances wherein two people agreed divided by the total opportunities where they could have agreed. Though this is easy to calculate, easy to communicate, and provides a balanced comparison of agreement relative to total opportunities, it does not provide specific information on disagreements, is misleading with imbalanced datasets, and has decreased corrective utility with ceiling/floor effects. In this manuscript, we describe eight alternative measures from the binary classification literature. For each measure, we describe its benefits and drawbacks and the conditions under which each might have greater practical utility based on the function of analyzing IOA data. Understanding the measures described in this manuscript may allow behavior analysts to get more information from their IOA data.