Answers are grounded in this paper and cite their sources.
Gradient-based training of a parametric model is the motion of a damped particle in the potential set by the loss. Using the Bateman–Galley doubling of the degrees of freedom, which gives dissipative systems an action principle, we write gradient flow, the heavy-ball method and Nesterov's accelerated gradient method as Hamiltonian systems. We show that, within this Hamiltonian description, backpropagation is Dirac evolution. The activations of a feed-forward network are treated as coordinates of phase space, and the layer equations as constraints. These constraints are second class for every architecture, and Hamilton's equations with the resulting Dirac bracket reproduce training with the exact gradient of the loss, although the Hamiltonian contains only the cost of the network output. The Dirac bracket of an activation with the momenta of the parameters is its Jacobian, the inverse of the constraint matrix contains both the forward and the reverse modes of automatic differentiation, and the errors propagated by backpropagation are the Lagrange multipliers of the layer constraints: the constraint forces of a kinematic chain under the load of the cost. These results hold in the full doubled phase space, not only in the physical limit. The network's part of the constraint algebra is the same for every optimizer, Nesterov's look-ahead adds one layer of momentum-dependent constraints that couple the two copies of the doubled system, and gradient flow is obtained by constructing the Dirac bracket in stages. We confirm these statements numerically to machine precision and train a network by integrating the Dirac flow of its constraints, with no backward pass in the computation.
No takes yet. Share an insight, caveat, or question.
Holanda et al. (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: