HPformer: Low-Parameter Transformer With Temporal Dependency Hierarchical Propagation for Health Informatics

Transformers based on Self-Attention (SA) mechanism have demonstrated unrivaled superiority in numerous areas. Compared to RNN-based networks, Transformers can learn the temporal dependency representation of an entire sequence in parallel, while efficiently dealing with long-range dependencies. However, the O(L²)O(L2) (LL denotes the length of the sequence) computational complexity of the SA mechanism and the high memory usage make the construction cost of the Transformer-based model prohibitively expensive. To address these challenges, we propose a Transformer-like model, HPformer: Low-Parameter Transformer with Temporal Dependency Hierarchical Propagation. HPformer first chunks the sequence into KK (K = log L + 1K=logL+1, ·· denotes ceiling operation) sequence segments, then leverages the hierarchical propagation mechanism with O(L)O(L) computational complexity to learn the temporal dependencies between the segments and within the segments, and ultimately generates KK vectors as $Key$Key matrices. This reduces the complexity of the SA mechanism from O(L²)O(L2) to O(Llog L)O(LlogL). In addition, we employ a strategy of sharing $Key$Key and $Value$Value matrices between layers to build the HPformer, thus reducing memory usage. Extensive experiments based on public health informatics benchmark and Long-Range Arena (LRA) benchmark have demonstrated that HPformer has advantages over Transformer-based models in terms of memory usage and efficiency.