The effect of sample size and bias on the reliability of estimates of error: a comparative study of Dahlberg's formula

This study examined the effects of different sample sizes and different levels of bias (systematic error) between replicated measurements on the accuracy of estimates of random error calculated using two common formulae: Dahlberg's and the 'method of moments' estimator (MME). Computer-based numerical simulations were used to generate clinically realistic measurements involving random errors with a known distribution. For each simulation, two sets of 'measured values' were generated to provide the replicated data necessary for the estimation of the random error. Dahlberg's and the MME formula were applied to these paired data sets and the resulting estimates of error compared with the 'true' error. Nine different sample sizes (n = 2, 5, 10, 15, 20, 25, 30, 50, and 100) and two different types of bias (additive and multiplicative) were examined for their effect on the estimated error. In each case, the estimates of the random error were based on the distribution of 5000 separate simulations. The results indicate that with a sample of less than 25-30 replicated measurements, the resulting estimates of error are potentially unreliable and may under or overestimate the true error, irrespective of the formula used in the calculation. Where, however, a bias exists between the replicate measurements, Dahlberg's formula can be expected to overestimate the true value of the random error even where the biases are small and difficult to detect by standard statistical tests. No such distorting effect was found for the MME formula, which provided estimates of error that were not meaningfully different from the true value even where relatively large biases existed between the replicates. These results suggest the following: 1. A sample of at least 25-30 cases should be replicated to provide an estimate of the random error. 2. Where the original study contains fewer than 20 cases, the estimate of error will be unreliable. In these circumstances, it would be helpful if a confidence interval for the true error was also quoted. 3. Unless one can be absolutely sure that no bias exists between the replicate measurements, Dahlberg's formula should be avoided and the MME formula used instead.