Using cluster-based permutation tests to estimate MEG/EEG onsets: how bad is it?

Localising effects in space, time and other dimensions is a fundamental goal of magneto- and electro-encephalography (EEG) research. A popular exploratory approach applies mass-univariate statistics followed by cluster-sum inferences, an effective way to correct for multiple comparisons while preserving high statistical power by pooling together neighbouring effects. Yet, these cluster-based methods have an important limitation: each cluster is associated with a unique p value, such that there is no error control at individual time points, and one must be cautious about interpreting when and where effects start and end. Sassenhagen & Draschkow (2019) provided an important reminder of this limitation. They also reported results from a simulation, suggesting that onsets estimated from EEG data are both positively biased and very variable. However, the simulation lacked comparisons to other methods. Here I report such comparisons in a new simulation, replicating the positive bias of the cluster-sum method, but also demonstrating that it performs relatively well, in terms of bias and variability, compared to other methods that provide point-wise p values: two methods that control the false discovery rate, and two methods that control the family-wise error rate (cluster-depth and maximum statistic methods). I also present several strategies to reduce estimation bias, including group calibration, group comparison, and using binary segmentation, a simple change point detection algorithm that outperformed mass-univariate methods in simulations. Finally, I demonstrate how to generate onset hierarchical bootstrap confidence intervals that integrate variability over trials and participants, a substantial improvement over standard group approaches that ignore measurement uncertainty.