A reinforcement learning model of adaptive persistence

Valuable future outcomes are not always worth waiting for indefinitely. People can adaptively calibrate their willingness to wait for temporally uncertain rewards on the basis of experience with the relevant statistical distribution of delay intervals. Laboratory experiments in humans have documented a broad pattern of adaptive calibration accompanied by substantial individual-level variation. However, the computational mechanisms that produce this behavior are not yet well characterized. Here, we developed and evaluated a theoretical framework for attributing variation in adaptive delay-tolerance to latent computational parameters. We constructed a reinforcement learning model that conceptualized persistence decisions as a series of covert wait-or-quit choices over the course of a continuous delay. We evaluated the model's performance using a previously published experimental data set and two new experiments. We found that the model could adaptively calibrate persistence in an asymptotically optimal manner. Participant-level parameter fits enabled the model to account for the range of behaviors seen in empirical data. A variant of the model using a Q-learning mechanism with valence-dependent learning rates outperformed variants that used R-learning or a single learning rate. Parameters of the model showed good recoverability and efficiently captured multiple dimensions of human behavioral variation, including variation in overall levels of persistence, rates of experience-driven adjustment, and levels of event-to-event behavioral stochasticity. Our findings provide a candidate theoretical framework to support investigations of variation in temporally extended decision behavior across individuals or populations.