Evaluation of artificial intelligence systems for assisting neurologists with fast and accurate annotations of scalp electroencephalography data

Figures

• 

  Fig. 1. A block diagram of the high-level architecture of the custom-built challenge platform that depicts data and model flow during challenge operation. In this model-to-data paradigm challenge participants at no point download or access the data directly. Instead they create and submit models to the platform (green solid arrows) which automatically organises training and testing and then provides feedback on model performance to participants (orange dashed arrows). This is fundamentally different to conventional crowdsourced challenge setups. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

• 

  Fig. 2. All 5 valid final submissions were tested against validation and blind test sets. The plots show the results for (a) evaluation metric (E), (b) sensitivity (S), (c) false alarm rate (FA/24 h and (d) sensitivity (S) plotted as a function of FA/24 h.

• 

  Fig. 3. In order to label 24 h of EEG recordings an unassisted human annotator has to review all 24 h of raw EEG data (top). Using the systems developed in this challenge, the amount of data needing review is the sum of the seizure ground truth (correctly detected true positive actual seizure segments) plus the annotation overhead (incorrectly detected false positive segments). All 4 automatic systems operate at 75% detection sensitivity. A conservative upper bound approximation for the total seizure ground truth duration in a 24 h raw EEG data recording is ~0.2% [22] or ~3 min. The best models achieve a minimum annotation overhead of 7 min which therefore allows to reduce the total amount of raw EEG data to be reviewed by a human annotator from 24 h down to 10 min or less. Note that the duration of seizure ground truth may fluctuate across patients, i.e. a patient might experience longer or more frequent seizure episodes on certain days which impacts the total duration of raw EEG data to be reviewed for that day. The annotation overhead however remains unaffected and will stay at the levels shown in the figure for all patients at all times.

• 

  Fig. 4. An engineering step introducing a hyperparameter which allowed a trade-off between sensitivity and FA rate was included in the submissions of teams Otameshi and Ids_comp. This engineering step was applied to all 5 final submissions 4 of which thereby reached sensitivities of 75% or higher. (a) shows false alarm rates at the 75% detection sensitivity mark for those 4 models. (b) shows the reduction factors of raw EEG data that has to be reviewed by human annotators for each system. Team EpiInsights achieves the highest reduction factor of 142x.

• 

  Fig. 5. FAs per 24 h plotted against detection sensitivity going from 75% sensitivity level to the maximum achievable sensitivity for each algorithm. The TAES metric causes the maximum achievable sensitivity for the model of team Ids_cpmp to stay below 80%.

• 

  Fig. 6. Reduction factors of raw EEG data to be reviewed by a human annotator vs. detection sensitivity going from 75% to maximum achievable sensitivity values for each system. The models from teams Otameshi, EpiInsights and Team SG achieve maximum detection sensitivities of 90.63%, 91.60%, and 91.57%, respectively and two-order of magnitude data reduction factors.

• 

  Fig. 7. (a) Applying the engineering step introduced by teams Otameshi and Ids_cpmp raises the maximum detection sensitivities to 90.63%, 91.60% and 91.57%, respectively. This comes at the cost of increased false alarm rates and decreased data reduction factors which are shown in (b). Note that even at maximum sensitivity level the lowest data reduction factor (22, Team SG) still allows to compress 24 h of raw EEG data down to a ~1h-short segment of raw EEG data to be reviewed by a human annotator.