Off-Policy Evaluation with Irregularly-spaced, Outcome-dependent Observation Times

While classic off-policy evaluation literature typically assumes evenly spaced decision times for analytical convenience, this assumption is often violated in real-world applications, such as those involving user-initiated visits.To accommodate irregularly-spaced and potentially outcome-dependent decision times in off-policy evaluation, we introduce a policy evaluation criterion defined through integrated rewards rather than the traditional cumulative-reward formulation.To ensure the estimability of the value function of a policy, we further develop a new reinforcement learning framework that incorporates not only the state-action process but also an observation process, where the gap times between observations can interact with states, actions, and rewards in a more flexible way.Statistical inference for the proposed policy value is established by positing a modulated renewal process model for the observation times.The validity of the proposed approach is further supported by theoretical analysis, simulation studies, and a real-world application that uses electronic health records to evaluate periodontal