The Mechanics of Neural Computation from High-Dimensional State Dynamics to Emergent Computation in Large Neural Networks

Neural computation is increasingly understood not as a sequence of isolated input–output operations, but as the structured transformation of high-dimensional internal states. This review develops a unified perspective on the mechanics of neural computation by tracing its evolution from early threshold-based and function-approximation models to recurrent dynamical systems, deep hierarchical networks, neural operators, and large Transformer architectures. The review examines how representation geometry, distributed representations, feature superposition, intrinsic dimensionality, residual transformations, recurrence, continuous-depth dynamics, and stability shape the evolution of neural states across depth and time. It further connects these principles to learned computational operators, physics-informed neural networks, Hamiltonian and variational architectures, differentiable simulators, and discretization-invariant neural operators for physical and scientific computation. At the scale of large neural networks, the review analyzes how features, memory-like components, attention mechanisms, and mechanistic circuits compose into structured computational processes, while training dynamics, scaling laws, in-context learning, and inference-time reasoning determine how these mechanisms emerge and are utilized. Across these perspectives, neural computation is characterized as an organized state-evolution process in which representation geometry determines how information is structured, learned operators determine how states are transformed, and mechanistic circuits determine how local computations compose into global behavior. The review also identifies unresolved challenges in computational efficiency, optimizer-dependent learning dynamics, the transition from memorization to structured generalization, and the distinction between behavioral reasoning and faithful internal computation, motivating the development of a more predictive mechanics of learned computation.