Revisiting Qualms about Bootstrap Confidence Intervals

SYNOPTIC ABSTRACTIn a rather important paper Schenker (1985) used a particular chi-square distribution for a sample variance to show that the percentile method bootstrap and even the BC bootstrap break down for very practical sample sizes. This caused Efron (1987) to devise the BCa method in a very well-known paper. This paper revisits the issues surrounding bootstrap confidence intervals by looking at a particularly difficult problem, estimating variances in a nonparametric setting. We show by simulation methods that for certain heavy-tailed and skewed sampling distributions for the observed data, the convergence of even the second order accurate bootstrap methods BCa. and ABC, must be slow because even at a sample size of nSize=200 the confidence level is not close to the advertised and correct asymptotic level (e.g. for Student's t with 5 degrees of freedom the methods compared are between 4 and 5% below their nominal levels). At nSize =25, all of the methods provide true confidence levels that are at least 5% below their nominal confidence levels. We illustrate this by using confidence levels of 50%, 75% and 90% among others. To investigate more deeply the convergence properties, we took nSize=1000 or more (i.e. number of observations in the original data set). To adequately show the pattern of convergence, the standard deviation of the Monte Carlo approximation of the proportion needs to be around 0.005. But to achieve this requires close to nRepI =10,000 iterations (i.e. Monte Carlo replications of the bootstrap estimates) of the simulated results! For the high order bootstraps at nSize=1000 and nRepl =10,000 computations become prohibitively intensive even on a fast modem computer! It is interesting that for these simulations the first order percentile method bootstrap did nearly as well as. and sometimes better than, the higher order bootstraps.