Driver cognitive load estimation in conditional driving with aligned attention-enabled multimodal fusion

Despite the promise of autonomous driving in enhancing road safety, drivers in conditionally automated vehicles are still responsible for driving safety and thus driver state still matters to driving safety. Though driving automation can reduce taskload of drivers, they may still experience high cognitive load, which can impair takeover performance. However, existing cognitive load estimation algorithms were primarily designed for non-automated vehicles, which may not be applicable in conditionally automated vehicles, due to the differences in driver responsibilities and the availability of certain metrics (e.g., driving performance measures are absent when drivers are not controlling the vehicle). Further, existing driver cognitive load algorithms rarely considered the integration of both spatial and temporal information in the input features. Therefore, we proposed an aligned-attention transformer network that integrates the multi-stream transformer network with alignment attention to estimate the cognitive load of drivers in conditionally automated vehicles. The algorithm fuses physiological measures that can potentially be measured non-intrusively in vehicles, i.e., electrocardiogram, electrodermal activity, and respiration signals. To validate the efficacy of the algorithm, we supplement a European dataset with a self-collected Chinese dataset, in which 42 drivers engaged in various cognitive tasks (i.e., memory, calculation, and spatial tasks). The results showed that our algorithm outperformed state-of-the-art driver cognitive estimation algorithms on both within-subject and across-subjects data partitions. Further, ablation tests validated the robustness of our algorithm and the effectiveness of the network modules. This research can guide the design of driver state monitoring systems in both non-automated vehicles and conditionally automated vehicles.