Seizure prediction: making mileage on the long and winding road

This scientific commentary refers to ‘Crowdsourcing reproducible seizure forecasting in human and canine epilepsy’, by Brinkmann et al. (doi: 10.1093/brain/aww045 ). Epilepsy is characterized by recurrent spontaneous seizures that seem to occur without warning, like a bolt from the blue. Most patients with epilepsy spend over 99.9% of their time in a perfectly normal brain state. But even though epileptic seizures constitute less than 0.1% of their lifetime, it is the apparent unpredictability of these events that puts an immense psychological burden on patients and substantially compromises their quality of life. A reliable method of forecasting seizure occurrence would thus greatly diminish psychological stress for patients. Moreover, it could lead to novel therapeutic options such as on-demand treatment and ultimately even closed-loop interventional devices to prevent impending seizures from occurring ( Morrell and Halpern, 2016 ). In this issue of Brain , Brinkmann and co-workers have used the concept of crowdsourcing to tackle the problem of seizure prediction ( Brinkmann et al. , 2016 ). They hosted epochs of continuous multi-day recordings of intracranial EEG (iEEG) activity from both humans and canines on a website ( www.kaggle.com ), and started a competition open to the public in which the aim was to correctly characterize these data clips as preictal (indicative of an impending seizure) or interictal (in this study, at least 1 week away from any seizure). Participating teams were provided with a set of training data consisting of clips that were labelled as being preictal or interictal. Using these data clips, they could train and optimize their algorithms before running them on the other clips provided—the testing data. For each of these remaining clips, the candidate algorithms had to estimate the probability that the clip was preictal. Receiver operating characteristics (ROC curves) were then used to assess the suitability of these probability estimates for distinguishing interictal from preictal clips. Each time the classification results were submitted to Kaggle, the submitting teams instantly received their classification score—given by the area under the ROC curve (AUC)—for 40% of the unlabelled testing data (‘public leader board’). Scores for the remaining 60% of testing data were used as the actual final ranking scores (‘private leader board’), but were not disclosed to the contestants until the end of the submissions period marking the end of the competition. The designers of this competition chose to address the question of whether seizures can be predicted by using a retrospective statistical analysis design, rather than by evaluating the performance of a (quasi) prospective algorithm required to issue a warning of an impending seizure (such as a red light). Nevertheless, relevant guidelines for a proper assessment ( Mormann et al. , 2007 ) were carefully followed in that (i) unselected datasets from long-term recordings were used, comprising a fairly representative sample of training and testing data for each human or canine subject; (ii) both sensitivity and specificity were accounted for by means of the ROC formalism; (iii) statistical significance of the AUC scores was assessed using the Hanley-McNeil method; and (iv) optimization of the algorithms was performed only on the labelled training data, while classification performance was strictly reported only for out-of-sample testing data. To account for a bias due to repeated submissions of adjusted algorithms (some teams submitted up to 500 versions of their algorithms) the designers of the competition evaluated the most successful algorithms in the private leader board using previously held-out data that had never been seen by any of the submitted algorithms. The final classification performance on these held-out datasets reported by Brinkmann et al. was thus completely out-of-sample and free of any in-sample optimization bias due to multiple testing. Remarkably, all six of the top 10 teams (based on scores on the private leader board), who chose to submit their algorithms for performance evaluation on the held-out data, achieved AUC scores that were significantly better than chance ( P < 0.0001 in all cases). The chance level in this study corresponded to a situation where all probability estimates were drawn at random from a given distribution. It is worth noting that five of these six algorithms employed both univariate and bivariate approaches, thus characterizing both local phenomena as well as interactions between different recording sites. The characterizing features were mostly linear measures in the frequency or time domain rather than non-linear measures ( cf . Mormann et al. , 2007 ). Shared databases for seizure prediction have some history in this field. For the ‘First International Workshop on Seizure Prediction’ held in 2002 in Bonn, Germany, a set of five continuous long-term recordings from five different epilepsy centres was shared among participants to test and compare their algorithms ( Lehnertz and Litt, 2005 ). Access to this dataset, however, was restricted at the time to the community of workshop participants. An important milestone in publically available datasets was a database published by the University of Freiburg, Germany, which consisted of 88 seizures from 21 patients ( Winterhalder et al. , 2003 ). Recordings in this database were not contiguous but instead included 50 min of preictal data per seizure and 24 h of interictal baseline data per patient. In addition they were restricted to three data channels from the seizure onset zone and three channels outside this zone. More recently, a multinational European project (EPILEPSIAE) resulted in the compilation of a database of 275 patients with continuous multi-day recordings of intracranial and/or scalp EEG, fully annotated and available in exchange for a maintenance fee by interested groups ( Klatt et al. , 2012 ). Around the same time, the NINDS-funded International Epilepsy Electrophysiology Portal (ieeg.org) was established as a cloud-based research platform with more than 1200 datasets of continuous scalp and intracranial EEG recordings from both animal models of epilepsy and patients ( Wagenaar et al. , 2015 ). Although dozens of studies on seizure prediction were published after the mid-1990s, it was not until 2011 that the first investigation provided statistically sound evidence that seizures can be predicted above chance level ( Park et al. , 2011 ). Although this study analysed the Freiburg dataset rather than continuous recordings, it is to the best of our knowledge the first publication to report algorithm performance for out-of-sample testing data and to assess its statistical significance. A further milestone in the field of seizure prediction was an Australian clinical trial in which a warning device was implanted in 11 selected patients for at least 4 months. In nine of these patients, a better-than-chance warning performance was achieved using a truly prospective algorithmic design ( Cook et al. , 2013 ). The iEEG recordings from this study were stored, and they represent a unique dataset of unprecedented length during normal, everyday conditions for the patients and under stable medication. This recording situation is similar to that of the canine data acquired by Brinkmann and colleagues and quite different from the conditions in an epilepsy monitoring unit. It would therefore be highly desirable if the recordings performed by Cook and colleagues (2013) could be made accessible to the research community, perhaps in the form of another Kaggle competition. The idea of a seizure prediction competition open to the public such as the one staged on Kaggle is not unprecedented. A similar competition was designed for the ‘Third International Workshop on Seizure Prediction’ held in Freiburg in 2007 ( Schelter et al. , 2010 ). However, in this competition none of the few algorithms submitted were able to convincingly demonstrate predictive performance above chance level. The secret to success for the competition featured by Brinkmann and co-workers may have been its high visibility on Kaggle. Whatever the reasons for success might have been, an exceptional merit for the scientific community must be seen in the fact that algorithms that succeeded in classifying the held-out data clips are available online. The availability of open data along with the availability of open source codes of successful algorithms will likely be a strong asset for further advances in the field of seizure prediction. Glossary: Hanley-McNeil method: This method translates the area under an ROC curve (AUC) into the probability that a randomly chosen item with a certain characteristic (here a preictal data clip) receives a higher rating or ranking (here a higher probability estimate of being preictal) than a randomly chosen item without this characteristic (here an interictal data clip). In-sample optimization: If an algorithm or classification scheme is optimized for a given set of training data, the achieved performance will likely not be reproducible when applied to novel data. Out-of-sample testing: After optimizing and fixing the parameters of an algorithm or classification scheme, its performance should be tested on novel, never-before-seen data to obtain a reliable estimate.