Prescreening Layer: Lightweight Key Modulation for Attention Mechanisms

Attention mechanisms have become ubiquitous in modern deep learning architectures, yet their default formulation treats all key dimensions uniformly across sequencepositions. We introduce the Prescreening Layer (PSL),a lightweight, learnable module that dynamically modulates key tensors prior to query–key interaction. PSL aggregates temporal information via mean pooling, passesthe result through a small two-layer MLP with a sigmoidgate, and applies the resulting modulation as a residual addition to the original keys. Our experiments ona controlled transformer setup show that a minimal PSLconfiguration (hidden dimension 4) improves final lossby 4.9% over the baseline with only 0.58% additionalparameters. Larger PSL configurations degrade performance, suggesting that highly compressed prescreeningis sufficient and beneficial. We provide detailed analysisof gate statistics, gradient flow, and computational overhead