Executable Matrix Programs: Faithful Weight-Derived Replacement and Factorized Intervention in Pretrained Transformers

We present a method for extracting executable component-level matrix programs from a pretrained decoder-only transformer without retraining or modifying its weights. For RoPE attention, the method constructs a weight-derived QK routing target Mqkᵃugd, indexed by relative distance, and a VO payload map Cᵥoᵃug; for a SwiGLU MLP, it constructs an exact atom program with a gate/read/write factorization. Unlike input-specific locally linear representations, these structural objects are derived directly from the weights and reused across inputs. On Qwen2. 5-0. 5B-Instruct, we simultaneously replace all 336 attention heads and all 24 MLP sublayers with program execution inside the native forward pass. Across 64 prompts, the median relative error of the full logit vector is 0. 258%, the 95th percentile is 0. 566%, the maximum is 0. 923%, and the top-1 prediction is preserved in 100% of cases. Using the same interface, we construct a complete factorized causal atlas: exact replacement, QK-off, and VO-off are measured separately for all 336 heads, while exact replacement and MLP-off are measured for all 24 MLPs. The median ratios of intervention effect to observed replacement discrepancy are 24. 89× for QK, 20. 62× for VO, and 152. 05× for MLPs. Downstream propagation profiles show that, for 75. 0% of QK interventions and 71. 4% of VO interventions, the peak internal effect occurs at least four layers after the source; the median delays to the peak are 10 and 9 layers, respectively. The matrix program therefore serves not only as an analytic description of a component, but as an executable interface for faithful replacement, factorized intervention, and analysis of distributed computation across model depth. Code and machine-readable results: https: //github. com/maxwelhelp/matrix-programs