Guidelines of the minimum sample size requirements for Kappa agreement test

Background and aims: To estimate sample size for Cohen’s kappa agreement test can be challenging especially when it is expected that the true marginal rating frequencies are not the same. This study aims to present the tables that display a minimum sample size determination for an agreement test when certain assumptions are hold. Method: We adopted the sample size formula provided by Flack and colleagues (1988) to calculate the minimum sample sizes required using PASS software. The power is pre-specified to be at least 80% and the alpha to be less than 0.05. The effect sizes were derived from several pre-specified estimates such as the pattern of the true marginal rating frequencies and the difference between the two kappa coefficients in the hypothesis testing. Results: When the true marginal rating frequencies are the same, the minimum sample size determination can range from 2 to 698 depending on the actual value of the effect size. When the true marginal rating frequencies are not the same, then the majority of the minimum sample size required for this condition is more than double than that required sample size when the true marginal rating frequencies are the same. Conclusion: Concerning that the sample size formula could produce a very extreme small sample size, therefore, the determination of K1 and K2 should be based on reasonable estimates. We recommend for all sample size determinations for Cohen’s kappa agreement test, the true marginal rating frequencies can be assumed the same. Otherwise, it will be necessary to multiple the estimated minimum sample size by two to accommodate if the true marginal rating frequencies is not the same.