Answers are grounded in this paper and cite their sources.
Learning systems run on rules that are fixed before they start: an optimizer and its schedule, an aggregation formula in federated learning, a detector for a monitored plant. Large language models (LLMs) that can read numerical state and write code now make it possible to inspect a process while it runs and to replace such a rule with a program written for that process. This survey examines that possibility along a five-stage loop of observation, agentic reasoning, program synthesis, verification and deployment. Methods are compared on four axes that cut across applications: the knowledge available to the LLM, the artifact it generates or modifies, the mechanism that validates it, and the cost and risk of using it. Three systems anchor the analysis: GradReason, which replaces training epochs with training-data-free weight updates; FedReason, which writes aggregation programs after reading client models; and PIA, which composes physical residual detectors for cyber-physical systems. They are read beside reactive optimizers, LLM-guided program search, learned optimizers, and non-agentic forms of knowledge reuse such as model annealing and knowledge fermentation. In the reported ablations, removing cheap empirical feedback costs far more than withholding model internals or documentation; agents mostly select and tune techniques that were known before their training cut-off; and once a working update is found, a fixed rule can recover most of the gain without inference cost. We derive a break-even condition for agent latency, propose a minimum evaluation protocol with budget-matched and rule-encoding controls, and set out open problems in discovery beyond prior knowledge, acceptance rules with long horizons, and synthesis under adversaries.
No takes yet. Share an insight, caveat, or question.
Ruzbahani et al. (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: