0320 Foundational Transformers with Linear Probing for Sleep Stage Classification Using Time Series Sleep Study Data

Abstract Introduction Sleep disorders and deprivation disrupt people's daily activities, mental health, and longevity and are related to widespread conditions. Currently, sleep disorders are diagnosed via polysomnography (PSG), where electrophysiological data is collected and manually annotated by a clinician. State of the art machine learning (ML) models, such as the transformer, are particularly well-suited for modeling timeseries PSG data. Specifically, self-supervised models with linear probing could assist with any relevant sleep predictive task including automating sleep stage classification, which would save clinicians time, reduce variability in manual scoring, and help scale to treat more people. Methods Using a self-supervised learning approach and the transformer architecture, we trained a self-supervised ML model that inputs seven PSG channels of length three hours including electroencephalogram, electrooculogram, electromyography, electrocardiography, oxygen saturation, and thoracic and abdomen respiratory rate using the Sleep Heart Health Study database (1995-1998). The model architecture uses the transformer’s attention mechanism to learn long range dependencies between intervals of sleep and a convolutional layer to learn relationships among channels. The model learns representations of PSG data through masked reconstruction with a mean squared error loss function. The representations are used as input into a deep neural network that is trained via linear probing (without adjusting the weights of the transformer model) to classify sleep stages. Results 5,794 sleep studies from the Sleep Heart Health Study with at least three hours of relevant PSG sleep channel and hypnogram data are included in training the self-supervised and linear probing models. Area under the receiver operator characters curve for sleep stage classification are 0.960 0.960-0.961, 0.848 0.846-0.849, 0.906 0.906-0.907, 0.968 0.967-0.968, and 0.931 0.930-0.932 for wake, stage 1, stage 2, stage 3, and REM, respectively. Hyperparameter tuning, class weighting, and dataset cleaning will be performed to increase classification results. Conclusion A self-supervised training approach using the transformer architecture with linear probing was utilized to learn multichannel PSG data representations. These representations were used as input into a downstream model to classify sleep stages accurately. Future work should be done to examine the capabilities of the self-supervised model representations for other predictive sleep tasks. Support (if any) NIH K25HL151912, NIH R01HL171813, NIH R21HL165320