Averaged Operators and the Slow Flow of Depth in Trained Transformers

The depth of a trained transformer is studied as a discrete dynamical system: each layer advances a hidden state, and the network’s prediction is the endpoint of the evolution. We show empirically that this evolution is well described by a slowly varying effective field. A band of consecutive layers is reproduced as an input–output map by repeated application of a single averaged operator, defined as the mean of the layers’ update fields, with every weight taken from the pretrained model and nothing fitted; a forty-eight-layer GPT-2 network is reproduced by as few as four to eight such operators. The mechanism is a separation of scales: each layer’s update is the sum of a slowly varying component shared by neighboring layers and a fast component that is nearly orthogonal from layer to layer, and averaging suppresses the fast component according to a quantitative law given in the text while preserving the slow one. The admissible averaging width is small in the first layers and grows with depth; there is a sharp obstruction in the first layers caused by a sign reversal of the layer fields. Resolving the slow drift to first and second order, as in higher-order finite elements, reduces the error and halves the number of operators required. The averaged operator is more computationally expensive to evaluate than the layers it replaces, so its purpose is analysis of the computation rather than compression. The construction reproduces across GPT-2 medium, large, and XL, with a preliminary check indicating partial transfer to OPT.