Query-Value Attention: Eliminating Keys from Transformer Self-Attention While Preserving Learning

Transformer architectures rely on three learned projections queries, keys, and values—to compute self-attention. This paper introduces Query-Value Attention (QVA), a simplified attention mechanism that removes the learned key projection and computes attention directly using query-value similarity. By allowing value representations to simultaneously serve as both addressing and retrieved representations, QVA reduces the number of learned attention projections from three to two while preserving the remainder of the Transformer architecture. Experiments on a GPT-style language model trained on WikiText-103 demonstrate stable optimization and coherent text generation, suggesting that effective self-attention can be learned without a separate key projection. QVA also reduces attention projection parameters and approximately halves KV-cache memory requirements during autoregressive inference. This work investigates whether simplified representational structures can preserve transformer learning dynamics while reducing architectural complexity.