Conductor-Worker Attention: A Data-Efficient Architecture for Small Language Models Dilip Pokhrel: Independent Researcher, Kathmandu, Nepal Standard Transformer models treat all attention heads equally, every head does the same job in parallel, which leads to redundant computations and wasted model capacity. This paper proposes Conductor-Worker Attention (CWA), a new attention mechanism that splits heads into two specialized roles: Conductors read the full context and decide what is important for each token, generating dynamic gating signals through a lightweight network. Workers perform the actual attention computation, but only after their queries, keys, and values have been filtered by the Conductor's gates, so they focus only on what the Conductor deemed relevant. A diversity loss is also applied during training to prevent the Conductor heads from all learning the same thing. Key results on a 200M parameter model trained on 10B tokens: Outperforms Pythia-160M on HellaSwag (+5.4%) despite using 30× less training data Matches or beats GPT-2 Medium (355M params, 43% larger) on reasoning benchmarks Visualization confirms the hierarchy actually works: the model routes entity-related tokens to specific worker heads and suppresses irrelevant ones automatically The core argument is that explicitly structuring who plans and who executes inside the attention mechanism allows the model to learn more efficiently from less data, a meaningful advantage for researchers without access to large compute budgets.
Dilip Pokhrel (Mon,) studied this question.