FRAE-S: A Self-Calibrating, Physics-Informed Advantage Estimator for Adaptive Temporal Credit Assignment in Actor–Critic Reinforcement Learning

FRAE-S: Fluid Resonance Advantage Estimation—Stable A Navier–Stokes Inspired, Self-Calibrating Advantage Estimator for Actor–Critic Reinforcement Learning FRAE-S (Fluid Resonance Advantage Estimation—Stable) is an experimental reinforcement learning framework that rethinks temporal credit assignment in actor–critic systems by replacing the fixed global smoothing parameter used in Generalized Advantage Estimation with a local, dynamics-aware propagation rule derived from the TD-error stream itself. The core idea is to interpret advantage estimation as a flow-control problem: the TD error is treated as an information signal, local variance is modeled as a viscosity-like quantity, and the rate of change in the signal is converted into an information Reynolds number that determines how strongly credit should be propagated backward through time. In this formulation, stable regions of the trajectory permit broader credit flow, while high-variance or sharply changing regions attenuate propagation, producing a more adaptive and interpretable estimator than a single static λ value. The method is built around three linked components. First, an exponentially smoothed variance estimate acts as a viscosity invariant, giving each timestep a scale-sensitive measure of local instability. Second, a Reynolds-style ratio compares the current TD-error momentum against that viscosity, allowing the estimator to distinguish laminar learning phases from turbulent ones. Third, a smooth Gaussian phase-transition gate converts that Reynolds number into a per-step credit coefficient, so the backward return recursion automatically becomes more conservative in unstable regions and more permissive in stable ones. FRAE-S also normalizes advantages intrinsically by dividing by the local viscosity term, which removes the need for separate batch advantage normalization and embeds stability directly into the estimator itself. A further contribution is the topological calibration protocol, which uses a short random-policy probe before training to estimate environment-native statistics and set key hyperparameters automatically. Instead of hand-tuning the viscosity floor, critical Reynolds threshold, and entropy coefficient, the probe derives them from observed signal scale and reward topology, aiming to reduce manual sensitivity and make the method more self-calibrating across tasks. In the reported LunarLander-v3 experiment, this calibration selected a dense-topology entropy setting and a nontrivial viscosity scale that matched the environment’s reward dynamics. The implementation is intentionally parameter-free at the estimator level and is layered on top of a standard decoupled A2C backbone. The actor and critic networks remain conventional multilayer perceptrons, while the FRAE-S engine operates as a deterministic transformation over rollout data, with linear time complexity in trajectory length. This makes the contribution primarily architectural rather than model-capacity-based: the novelty lies in how credit is assigned, stabilized, and interpreted, not in adding extra learnable modules. Empirically, the manuscript reports a single-environment, single-seed validation on LunarLander-v3 over 750,000 interactions. The logged telemetry shows a transition from turbulent exploration to a laminar mid-training phase and then to high-performance turbulent behavior later in training, with the reward eventually reaching a peak of +238.3 and crossing the +200 benchmark at epoch 1050. The paper presents these results as promising and interpretable, while also emphasizing that they are preliminary and not yet supported by the kind of multi-seed, multi-environment, and ablation-based validation needed for strong claims.