TCBI: A Three-Channel Behavioral Image-Based Method for encrypted malicious traffic detection

The widespread application of encrypted traffic, while protecting user privacy, also provides a covert communication channel for malware. Existing machine learning-based detection methods suffer from insufficient feature representation capability and weak generalization performance when facing frequent malware variants and unknown attacks. To address this issue, this paper proposes TCBI (Three-Channel Behavioral Image). This method first proposes a novel encrypted traffic representation approach, which reconstructs network sessions into tri-channel behavioral images. According to the distribution characteristics of different features, nonlinear or piecewise mapping strategies are designed to preserve information completeness while enhancing the discriminability of key behavioral features, thereby effectively capturing the structural similarity and behavioral homology of attack traffic. Secondly, a dual-branch parallel detection model combining Swin Transformer and Bi-LSTM is designed. The image branch extracts local details and global structures of behavioral images through a window-based self-attention mechanism, while the sequence branch utilizes a Bi-LSTM to mine the temporal dependencies of packet length and inter-arrival time, enabling collaborative modeling of spatial and temporal features and improving the accurate identification capability of malicious traffic. Finally, learnable adaptive weighted fusion mechanisms are introduced both within the sequence and between the image and sequence, dynamically adjusting the contribution weights of different feature branches. This allows the model to flexibly integrate multi-modal information according to sample characteristics, significantly enhancing its generalization ability and robustness against variant and unknown malicious traffic. Experimental evaluation results show that TCBI achieves excellent performance in known malicious traffic detection and demonstrates superior generalization ability and robustness in unknown malicious traffic detection, outperforming existing state-of-the-art methods.