This paper presents Layer-wise Local Supervision (LLS) — a training method for deep neural networks built on a direct observation about how these models compute: every layer of a Transformer performs a matrix multiplication that transforms one vector into a slightly more refined version of the same vector, advancing it incrementally toward the final prediction. Since each intermediate representation already carries information about the input, it can be evaluated against the target using a shared classification head. This makes it possible to assign a clear local objective to every layer — independently of what other layers are doing — and to train each window of consecutive layers with its own gradient, without retaining activations from the rest of the network. Experiments on the text8 language modelling benchmark show that LLS with a fixed sliding window of size 4 scales favourably with depth: a 128-layer model trained with LLS outperforms standard backpropagation in 95.4% of training batches, with a mean loss advantage of +0.98 and a final loss improvement of 13.4%. The method reduces activation memory from O(N·d) to O(w·d), making training independent of network depth with respect to memory. Gradient chains are bounded by the window size, structurally eliminating vanishing and exploding gradient risk regardless of how deep the network is. LLS is architecture-agnostic in the sense that it applies to any model composed of repeated layers followed by a final prediction head — Transformer encoders, decoders, and in principle any deep homogeneous stack.
Kostiantyn Azarenko (Fri,) studied this question.