Making do with what we have: use your bootstraps

Bootstrap methods are a recent innovation They depend on adequate computing power Bootstraps are more empirical than other statistical processes Given a suitable sample, many population features can be derived from the sample alone The process takes repeated random samples from the original data The process is flexible and has many applications A jack knife is a pocket knife that is put to many tasks, because it's ready to hand. Often there could be a better tool for the job, such as a screwdriver, a scraper, or a can-opener, but these are not usually pocket items. In statistical terms, the expression implies making do with what's available. Another simile, of an extreme situation, is the bootstrap: extricating oneself from a predicament by the only means available (Curran-Everett, 2009). These everyday terms have been applied to statistical methods that allow us to work with limited data and draw robust conclusions. We have already discussed permutation tests, which are similar in that they use only the data collected, and make no assumptions about the population (Drummond it can also be used for comparisons, but other tests such as permutation tests may be more appropriate here. Let us compare a sample of frogs from the north of California, with a group sampled near Calaveras, where escapees from the jumping competition have interbred with native frogs. We wish to compare estimates of the two populations from which these samples have been drawn. We generate repeated random pseudo-samples from each sample. Each time a value is taken, we can choose any of the values that are in the original sample so there is the opportunity for each value to be chosen more than once, in each pseudo-sample, and from pseudo-sample to pseudo-sample. (This is sampling ‘with replacement’, i.e. after it has been chosen, the value is replaced in the original stock of values.) Figure 2 shows a simplified version of the process. The mean of each pseudo-sample is calculated. Figure 2 shows the first six of these pseudo-samples as dot plots. If we continue taking pseudo-samples until we have 20, and calculate the mean of each of these, we can then arrange these mean values as a distribution histogram, as seen in the bottom panels of Fig. 2. By taking the central 18 values we define the 90% confidence limits for the mean of the original sample. In practice, the process is repeated many more times than this. Typically we could generate 10,000 pseudo-samples to generate a range of mean values. We use these mean values to generate the confidence limits. This is referred to as the bootstrap percentile method. The distribution of the means for this number of samples can be seen in Fig. 3. The distribution of the means derived from the repeated pseudo-samples Further analysis of the data can be done with the same basic method. We can use the method to conduct comparisons. Thus in our example we find that the mean jump distance of our Calaveras sample is 0.44 m greater: what are the 95% confidence intervals of this estimate? We approach this by independently drawing a random pseudo-sample, with replacement, from each group (as in Fig. 2B and C), and calculate the mean value of each sample. The difference between these mean values is an estimate of the difference between the groups. We continue to repeat the process of randomly taking pseudo-samples from each group and calculating differences between the means. Figure 4 shows the distribution of differences obtained after repeating this process 10,000 times. If we count 250 from each extreme of these differences we define the 95% confidence interval for the original observed difference: 0.44 m (0.039, 0.847). These values do not include zero, and thus we conclude that the observed difference in jump distance between the two groups of frogs is unlikely to be a result of chance, and gain an indication of the likely size of this difference. The distribution of the differences of the mean values obtained from pseudo-samples taken from each sample The dashed lines indicate the central 95% of these values and the continuous line indicates zero difference. The bootstrap method is flexible and robust, well suited for analysis of data whose population distribution is uncertain, as is often the case in biological studies (Efron & Tibshirani, 1993). For example, assumptions about distribution can be avoided when comparing quanta at synapses (Van der Kloot, 1996). There may be occasions when the bootstrap can fail: for example it is not good with extreme distributions, or to estimate statistics – like the maximum – that depend on very small features of the data. Modern computers make the tedious procedure of repeated sampling straightforward. However, standard textbooks and the standard statistics packages have failed to acknowledge the value of the bootstrap approach. Curran-Everett describes how to use the statistical software package R for bootstrap methods (Curran-Everett, 2009), an add-on facility exists for SPSS and Excel, and Cole has described a macro to use with SAS (Cole, 1999). Most current packages lack standard facilities for these procedures. These useful and powerful methods should become gradually more common in standard statistical software. Note: Those with some programming skills can set up basic approaches themselves. Useful insights on how to implement different bootstrap methods in various programming languages can be found in Good's book (Good, 2006). For users of the Python programming language, the data used in this paper and the code used for its analysis is available at http://bit.ly/KJ67RW (Calmettes, 2012).