Post-Training Linearization of Neural Networks: From Nonlinear Activations to Discrete Linear Decision Structures

We show that for any trained neural network with pointwise nonlinear activation functions, each activation y = f(x) can be rewritten exactly as a per-input scalar gain g = f(x)/x absorbed into the adjacent weight matrix, replacing the activation with identity. The resulting network—an affine cascade selected by the gain pattern—produces identical output (logit difference = 0), verified across five CNN architectures (ResNet-50, VGG-16, DenseNet-121, MobileNetV3-Large, ConvNeXt-Tiny) covering ReLU, Hardswish, Hardsigmoid, and GELU. This rewriting does not remove nonlinearity; it relocates it into the gain pattern, where its information content can be measured: 4–5 effective bits per activation, consistent with the quantization literature. Inference-time nonlinearity is thus a low-bit discrete routing signal, and a trained CNN a cascaded discrete linear decision structure. The gain lens exposes structure that no existing importance measure can see. In MobileNetV3Large—an architecture where removing 5% of channels collapses accuracy to near zero—gain statistics show that 30.7% of activations operate at constant gain and can be replaced by fixed scalars at a 0.5% held-out accuracy cost, with a further 0.2% dead and removable loss lessly; magnitude and Taylor criteria cannot distinguish these channels from important ones. Further consequences include complete causal decision path tracing, single-channel error diagnosis that restores 17% of misclassifications when the true label is given, and exact bidirectional input/output attribution per channel