Homeostasis Revisited and Reformulated Through Hidden Markov Model Control

A common formalization of homeostasis is the free energy principle, a framework that defines a set of desired observation values, or critical states, that the agent should reach or remain close to. Under the free energy principle, an agent should act to maximize the probability of receiving the desired observations. Maximizing this probability pushes the agent to avoid surprising events, those that put it at risk of homeostatic breakage. Here we revisit the common approach of solving the problem of maximizing the log probability of the desired observations by maximizing a variational lower bound, the so-called negative free energy. We show that, instead, directly maximizing that probability under the agent’s policy is better suited to, and provides a better solution for, the original homeostatic control problem. This is done using hidden Markov model (HMM) control by allowing the policy to act over hidden states or noisy versions thereof while trying to maximize the probability of repeatedly having the desired observations. HMM control substantially improves performance over variational inference. We also show that there is a deterministic optimal policy associated with HMM control, while variational inference yields a stochastic policy approximation. HMM control also generalizes classical results on probabilistic reachability and safety within constrained regions. We finally provide a homeostatic reinterpretation of the maximum occupancy principle—a principle proposing that agents ought to maximize the occupancy of action-state path space—by defining homeostatic states as any states that do not immediately entail the termination or death of the agent.