Modular Connections: The Connections Between Residual Streams Can Themselves Be Modular Models

Manifold-Constrained Hyper-Connections (mHC) widen the residual stream of a Transformer into N parallel streams and, at every sublayer, generate per-token coefficients that decide which streams to read from, which streams to write to, and how the streams mix. I show that once the N streams are interleaved, these three operations are exactly a convolution and a deconvolution with kernel size and stride N, and a sliding N×N map, so the connections of each token are simply a set of kernels. Building on this view, I propose Modular Connections (MoC): every sublayer keeps banks of read, write and mixing experts, and each token is routed to a few of them, as in a mixture of experts, and combines them with weights. This design matches a recent analysis of DeepSeek-V4-Flash, whose trained mHC settles on fairly fixed connection patterns per sublayer with little per-token variation. Because the mixing matrices of MoC are expert parameters rather than per-token outputs, their Sinkhorn normalization can be precomputed after training. In my measurements on a 1.36B-parameter model, the MoC forward pass is 2.2–3.2× faster than that of mHC on single sequences without compilation and about 1.5× faster on short inputs once compiled, the training step compiles about 16× faster, and routing can be inspected expert by expert. In an ongoing 49B-token run, the validation bits-per-byte of MoC has stayed within −0.0009 to +0.0017 of mHC at matching steps from step 2,000 to step 6,360. Web version with interactive animations: https://selen-suyue.github.io/modular-connections/en/