Incorporating Bayesian Networks in Markov Decision Processes

This paper presents an extension to a partially observable Markov decision process so that its solution can take into account, at the beginning of the planning, the possible availability of free information in future time periods. It is assumed that such information has a Bayesian network structure. The proposed approach requires a smaller computational effort than the classical approaches used to solve dynamic Bayesian networks. Furthermore, it allows the user to (1) take advantage of prior probability distributions of relevant random variables that do not necessarily have a direct causal relationship with the state of the system; and (2) rationally take into account the effects of accidental or rare events (such as seismic activities) that may occur during future time periods of the planning horizon. The methodology is illustrated through an example problem that concerns the optimization of inspection, maintenance, and rehabilitation strategies of road pavement over a 14-year planning horizon.