Development of a Model for Malware Detection and Classification at the Byte Level Based on Transformer

Malware threats have been a critical concern in cybersecurity, particularly due to the increasing complexity and constantly evolving variants that are difficult to detect using conventional signature-based or static rule-based methods. This research focused on developing a Transformer-based model at the byte level to detect and classify malware effectively and adaptively, thereby streamlining the analytical process without requiring specialized feature tokenization. The primary objective was to design and evaluate a Transformer model that captures universal and adaptive malware patterns directly from raw byte representations, enabling cross-platform applicability. A quantitative experimental approach was employed using three public datasets: Malware Detection PE-Based Analysis, MC-dataset-binary, and Malware.zip. Data processing involved byte embedding, dilated 1D convolution, multi-head self-attention, and attention pooling. Model optimization was conducted using AdamW with a combined scheduler, Stochastic Weight Averaging (SWA), random byte masking augmentation, and Mixup Embedding. Experimental results showed that the byte-level Transformer model achieved high classification accuracy across the three datasets, namely 99%, 92%, and 94%, respectively. These results demonstrate that a byte-level Transformer can effectively capture universal malware patterns in binary data, offering a flexible and highly accurate approach to developing resilient defenses against modern cyber threats. ABSTRAK: Ancaman perisian perosak (malware) merupakan isu kritikal dalam keselamatan siber, terutama apabila disebabkan oleh peningkatan kerumitan dan variasi yang sentiasa berevolusi, sekaligus menyukarkan pengesanan menggunakan kaedah konvensional berasaskan tandatangan atau peraturan statik. Kajian ini memfokuskan kepada pembangunan model berasaskan Transformasi pada peringkat bait (byte-level) bagi mengesan dan mengklasifikasikan perisian perosak secara berkesan dan adaptif, tanpa memerlukan pengekstrakan atau penandaan ciri khusus. Objektif utama kajian adalah bagi mereka bentuk dan menilai keberkesanan model Transformasi yang mampu menangkap corak perisian perosak bersifat universal dan adaptif secara langsung daripada representasi bait mentah, seterusnya membolehkan kebolehgunaan merentas platform. Pendekatan eksperimen kuantitatif digunakan dengan melibatkan tiga set data awam, iaitu Analisis Berdasarkan Pengesanan Perisian Perosak PE, Set Data Binari MC, dan Malware.zip. Pemprosesan data merangkumi pembenaman bait, konvolusi 1D terlarut (dilated), perhatian kendiri berbilang kepala, serta pengumpulan berasaskan perhatian. Pengoptimuman model dilaksanakan menggunakan AdamW dengan penjadual gabungan, Purata Pemberat Stokastik (SWA), penambahan data melalui penyamaran bait rawak, dan Penggabungan Embedding. Dapatan kajian melalui eksperimen menunjukkan bahawa model Transformasi peringkat bait mencapai ketepatan pengelasan tertinggi, iaitu masing-masing 99%, 92%, dan 94% bagi ketiga-tiga set data. Dapatan ini membuktikan bahawa Transformasi peringkat bait berupaya menangkap corak perisian perosak universal daripada data binari, sekaligus menawarkan pendekatan fleksibel dan berketepatan tinggi bagi membangunkan pertahanan yang lebih berdaya tahan terhadap ancaman siber moden.