Models of morality

Moral dilemmas engender conflicts between two traditions: consequentialism, which evaluates actions based on their outcomes, and deontology, which evaluates actions themselves. These strikingly resemble two distinct decision-making architectures: a model-based system that selects actions based on inferences about their consequences; and a model-free system that selects actions based on their reinforcement history. Here, I consider how these systems, along with a Pavlovian system that responds reflexively to rewards and punishments, can illuminate puzzles in moral psychology. Moral dilemmas engender conflicts between two traditions: consequentialism, which evaluates actions based on their outcomes, and deontology, which evaluates actions themselves. These strikingly resemble two distinct decision-making architectures: a model-based system that selects actions based on inferences about their consequences; and a model-free system that selects actions based on their reinforcement history. Here, I consider how these systems, along with a Pavlovian system that responds reflexively to rewards and punishments, can illuminate puzzles in moral psychology. Is it morally permissible to kill one person to save five others? Moral dilemmas like this engender conflicts between two major traditions in normative ethics. Consequentialism judges the acceptability of actions based on their outcomes, and therefore supports killing one to save five; ceteris paribus, five lives are better than one. By contrast, deontology judges the acceptability of actions according to a set of rules; certain actions (e.g., killing) are absolutely wrong, regardless of the consequences. Recent work has shown that experimental manipulations can sway people's judgments toward either consequentialism or deontology, suggesting that these perspectives have distinct neural underpinnings 1Greene J.D. The cognitive neuroscience of moral judgment.in: Gazzaniga M. The Cognitive Neurosciences. 4th edn. MIT Press, 2009: 987-999Google Scholar. One influential account of these findings posits that deontological judgments stem from automatic emotional processes, whereas consequentialist judgments result from controlled cognitive processes 1Greene J.D. The cognitive neuroscience of moral judgment.in: Gazzaniga M. The Cognitive Neurosciences. 4th edn. MIT Press, 2009: 987-999Google Scholar. Others argue that this dual-process approach is computationally insufficient and cannot explain how hypothetical scenarios are transformed into mental representations of actions and outcomes 2Mikhail J. Universal moral grammar: theory, evidence and the future.Trends Cogn. Sci. 2007; 11: 143-152Abstract Full Text Full Text PDF PubMed Scopus (548) Google Scholar. Universal moral grammar offers a computational theory of problem transformation, but lacks a neurobiologically plausible, mechanistic description of how values are assigned to mental representations of actions and outcomes, and how those values are integrated to produce a final consequentialist or deontological judgment. Recent advances in neuroscience offer a fresh perspective. Evaluations of actions and outcomes are guided by distinct decision-making systems that are psychologically and neurally dissociable 3Balleine B.W. O’Doherty J.P. Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action.Neuropsychopharmacology. 2009; 35: 48-69Crossref Scopus (1182) Google Scholar, 4Huys Q.J.M. et al.Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding.PLoS Comput. Biol. 2011; 7: e1002028Crossref PubMed Scopus (224) Google Scholar, 5Wunderlich K. et al.Mapping value based planning and extensively trained choice in the human brain.Nat. Neurosci. 2012; 15: 786-791Crossref PubMed Scopus (218) Google Scholar, 6Dayan P. How to set the switches on this thing.Curr. Opin. Neurobiol. 2012; 22: 1068-1074Crossref PubMed Scopus (66) Google Scholar. The model-based system generates a forward-looking decision tree representing the contingencies between actions and outcomes, and the values of those outcomes. It evaluates actions by searching through the tree and determining which action sequences are likely to produce the best outcomes. Model-based tree search is computationally expensive, however, and can become intractable when decision trees are elaborately branched. The computationally simple model-free system does not rely on a forward model. Instead, it evaluates actions based on their previously learned values in specific contexts (states): good state-action pairs are those that have produced desirable outcomes in the past (e.g., push door), whereas bad state-action pairs are those that have produced undesirable outcomes in the past (e.g., push person). Because the model-free system lacks access to current action–outcome links, it is retrospective rather than prospective and can make suboptimal recommendations in settings where traditionally good actions lead to undesirable outcomes, or vice versa 3Balleine B.W. O’Doherty J.P. Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action.Neuropsychopharmacology. 2009; 35: 48-69Crossref Scopus (1182) Google Scholar, 6Dayan P. How to set the switches on this thing.Curr. Opin. Neurobiol. 2012; 22: 1068-1074Crossref PubMed Scopus (66) Google Scholar. A third, Pavlovian system promotes automatic reflexive approach and withdrawal responses to appetitive and aversive stimuli, respectively 6Dayan P. How to set the switches on this thing.Curr. Opin. Neurobiol. 2012; 22: 1068-1074Crossref PubMed Scopus (66) Google Scholar. Pavlovian biases can influence behaviors guided by model-based and model-free evaluations: for example, in aversive Pavlovian-to-instrumental transfer, aversive predictions can suppress instrumental actions 4Huys Q.J.M. et al.Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding.PLoS Comput. Biol. 2011; 7: e1002028Crossref PubMed Scopus (224) Google Scholar. Pavlovian biases can also influence model-based evaluations themselves: searching a decision tree can be conceptualized as a set of internal actions that can be suppressed by aversive predictions 6Dayan P. How to set the switches on this thing.Curr. Opin. Neurobiol. 2012; 22: 1068-1074Crossref PubMed Scopus (66) Google Scholar. This amounts to a ‘pruning’ of the decision tree, whereby model-based tree search is curtailed when an aversive outcome is encountered 7Huys Q.J.M. et al.Bonsai trees in your head: how the Pavlovian system sculpts goal-directed choices by pruning decision trees.PLoS Comput. Biol. 2012; 8: e1002410Crossref PubMed Scopus (228) Google Scholar. There is now substantial evidence that model-based, model-free, and Pavlovian systems are situated in at least partly distinct brain circuits, although behavioral outputs likely reflect their combined influence 3Balleine B.W. O’Doherty J.P. Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action.Neuropsychopharmacology. 2009; 35: 48-69Crossref Scopus (1182) Google Scholar, 5Wunderlich K. et al.Mapping value based planning and extensively trained choice in the human brain.Nat. Neurosci. 2012; 15: 786-791Crossref PubMed Scopus (218) Google Scholar, and recent evidence suggests that certain regions integrate model-based and model-free evaluations 8Daw N.D. et al.Model-based influences on humans’ choices and striatal prediction errors.Neuron. 2011; 69: 1204-1215Abstract Full Text Full Text PDF PubMed Scopus (1000) Google Scholar. These systems often arrive at similar conclusions about the best action to take, but they sometimes disagree. Understanding how such conflicts are resolved is an active topic of research 6Dayan P. How to set the switches on this thing.Curr. Opin. Neurobiol. 2012; 22: 1068-1074Crossref PubMed Scopus (66) Google Scholar. On the surface, consequentialism and deontology appear to map directly onto model-based and model-free systems, respectively. Consequentialist and model-based approaches both evaluate actions based on their outcomes, whereas deontological and model-free approaches both evaluate the actions themselves. However, a deeper analysis reveals that deontological judgments likely arise from sophisticated interactions between systems. Consider one puzzle. In the classic trolley dilemma, a trolley is hurtling out of control down the tracks toward five workers, who will die if you do nothing. You and a large man are standing on a footbridge above the tracks. In one variant of this dilemma (trapdoor), you can flip a switch that will release a trapdoor, dropping the large man onto the tracks, where his body will stop the trolley. Is it morally permissible to flip the switch, killing the one man but saving the five workers? In another variant (push), you can push the large man off the footbridge onto the tracks, where again his body will stop the trolley. Is it morally permissible to push the man, killing him but saving the five workers? Intriguingly, when ordinary people confront these dilemmas, they are much less likely to endorse harming one to save five in cases in which harm involves physical contact with the victim (like pushing) than in cases in which harm does not involve physical contact (like releasing a trapdoor) 1Greene J.D. The cognitive neuroscience of moral judgment.in: Gazzaniga M. The Cognitive Neurosciences. 4th edn. MIT Press, 2009: 987-999Google Scholar, 2Mikhail J. Universal moral grammar: theory, evidence and the future.Trends Cogn. Sci. 2007; 11: 143-152Abstract Full Text Full Text PDF PubMed Scopus (548) Google Scholar, even though these cases have identical outcomes. Understanding how different decision systems evaluate actions and outcomes can illuminate puzzles such as the push–trapdoor divergence (Box 1). Consistent with universal moral grammar accounts 2Mikhail J. Universal moral grammar: theory, evidence and the future.Trends Cogn. Sci. 2007; 11: 143-152Abstract Full Text Full Text PDF PubMed Scopus (548) Google Scholar, I propose that the model-based system transforms hypothetical scenarios into a structural description of actions and outcomes (i.e., a decision tree). By searching the tree, the model-based system evaluates all possible outcomes and recommends the action that leads to the best outcome.Box 1Explaining aversion to physical harmsPrevious work attributes deontological judgments to automatic emotional processes 1Greene J.D. The cognitive neuroscience of moral judgment.in: Gazzaniga M. The Cognitive Neurosciences. 4th edn. MIT Press, 2009: 987-999Google Scholar. Here, I distinguish between retrospectively rational (but re-trainable) model-free mechanisms and ecologically rational fixed Pavlovian mechanisms, both of which sway judgments toward deontology in cases in which harm involves physical contact.The model-free system evaluates contextualized actions on the basis of their reinforcement history. Young children learn through experience that actions that physically harm others (e.g., hitting, pushing) result in aversive outcomes (e.g., punishments, distress cues 9Cushman F. et al.Simulating murder: the aversion to harmful action.Emotion. 2012; 12: 2-7Crossref PubMed Scopus (180) Google Scholar, 10Blair R. A cognitive developmental approach to morality: investigating the psychopath.Cognition. 1995; 57: 1-29Crossref PubMed Scopus (1041) Google Scholar). Simultaneously, parents and society verbally instruct children that physical harm is forbidden, warning about the consequences of transgressions. Both experience and instruction enable the model-free system to attach negative value to harmful physical actions toward people, as in a class of algorithms (called Dyna) that complement experiential trial-and-error learning with hypothetical trial-and-error learning 6Dayan P. How to set the switches on this thing.Curr. Opin. Neurobiol. 2012; 22: 1068-1074Crossref PubMed Scopus (66) Google Scholar. Importantly, the latter method, whereby model-free values can be retrained by model-based simulations, provides a route via which characteristically deontological judgments could be adaptive to changes in the environment that are detected by model-based mechanisms.By contrast, the Pavlovian system triggers responses to predictions of valenced stimuli; whereas the values of stimuli can be learned, Pavlovian proclivities to approach (avoid) appetitive (aversive) stimuli are fixed, like reflexes. For aversive predictions, one type of Pavlovian response is behavioral suppression; this response is ecologically rational in the sense that refraining from action is generally a good strategy when some actions might produce aversive outcomes 4Huys Q.J.M. et al.Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding.PLoS Comput. Biol. 2011; 7: e1002028Crossref PubMed Scopus (224) Google Scholar. Aversive predictions embedded within moral dilemmas could evoke Pavlovian processes that disfavor active responses, leading to characteristically deontological judgments. Harmful actions that involve physical contact may generate particularly strong aversive predictions (e.g., fearful expressions, screams, gore).Computational approaches to decision-making account for choices by adding up model-based, model-free, and Pavlovian action values, and then converting those values into action probabilities using a softmax function 4Huys Q.J.M. et al.Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding.PLoS Comput. Biol. 2011; 7: e1002028Crossref PubMed Scopus (224) Google Scholar, 6Dayan P. How to set the switches on this thing.Curr. Opin. Neurobiol. 2012; 22: 1068-1074Crossref PubMed Scopus (66) Google Scholar, 7Huys Q.J.M. et al.Bonsai trees in your head: how the Pavlovian system sculpts goal-directed choices by pruning decision trees.PLoS Comput. Biol. 2012; 8: e1002410Crossref PubMed Scopus (228) Google Scholar, essentially treating the three systems as separate experts, each of which ‘votes’ for its preferred action. Differences in state-action reinforcement histories (which influence model-free values) and aversive predictions (which influence Pavlovian values) could result in more ‘votes’ for inaction in the push scenario (Figure ID) than in the trapdoor scenario (Figure IC), leading to a higher proportion of deontological judgments in the former than in the latter. Previous work attributes deontological judgments to automatic emotional processes 1Greene J.D. The cognitive neuroscience of moral judgment.in: Gazzaniga M. The Cognitive Neurosciences. 4th edn. MIT Press, 2009: 987-999Google Scholar. Here, I distinguish between retrospectively rational (but re-trainable) model-free mechanisms and ecologically rational fixed Pavlovian mechanisms, both of which sway judgments toward deontology in cases in which harm involves physical contact. The model-free system evaluates contextualized actions on the basis of their reinforcement history. Young children learn through experience that actions that physically harm others (e.g., hitting, pushing) result in aversive outcomes (e.g., punishments, distress cues 9Cushman F. et al.Simulating murder: the aversion to harmful action.Emotion. 2012; 12: 2-7Crossref PubMed Scopus (180) Google Scholar, 10Blair R. A cognitive developmental approach to morality: investigating the psychopath.Cognition. 1995; 57: 1-29Crossref PubMed Scopus (1041) Google Scholar). Simultaneously, parents and society verbally instruct children that physical harm is forbidden, warning about the consequences of transgressions. Both experience and instruction enable the model-free system to attach negative value to harmful physical actions toward people, as in a class of algorithms (called Dyna) that complement experiential trial-and-error learning with hypothetical trial-and-error learning 6Dayan P. How to set the switches on this thing.Curr. Opin. Neurobiol. 2012; 22: 1068-1074Crossref PubMed Scopus (66) Google Scholar. Importantly, the latter method, whereby model-free values can be retrained by model-based simulations, provides a route via which characteristically deontological judgments could be adaptive to changes in the environment that are detected by model-based mechanisms. By contrast, the Pavlovian system triggers responses to predictions of valenced stimuli; whereas the values of stimuli can be learned, Pavlovian proclivities to approach (avoid) appetitive (aversive) stimuli are fixed, like reflexes. For aversive predictions, one type of Pavlovian response is behavioral suppression; this response is ecologically rational in the sense that refraining from action is generally a good strategy when some actions might produce aversive outcomes 4Huys Q.J.M. et al.Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding.PLoS Comput. Biol. 2011; 7: e1002028Crossref PubMed Scopus (224) Google Scholar. Aversive predictions embedded within moral dilemmas could evoke Pavlovian processes that disfavor active responses, leading to characteristically deontological judgments. Harmful actions that involve physical contact may generate particularly strong aversive predictions (e.g., fearful expressions, screams, gore). Computational approaches to decision-making account for choices by adding up model-based, model-free, and Pavlovian action values, and then converting those values into action probabilities using a softmax function 4Huys Q.J.M. et al.Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding.PLoS Comput. Biol. 2011; 7: e1002028Crossref PubMed Scopus (224) Google Scholar, 6Dayan P. How to set the switches on this thing.Curr. Opin. Neurobiol. 2012; 22: 1068-1074Crossref PubMed Scopus (66) Google Scholar, 7Huys Q.J.M. et al.Bonsai trees in your head: how the Pavlovian system sculpts goal-directed choices by pruning decision trees.PLoS Comput. Biol. 2012; 8: e1002410Crossref PubMed Scopus (228) Google Scholar, essentially treating the three systems as separate experts, each of which ‘votes’ for its preferred action. Differences in state-action reinforcement histories (which influence model-free values) and aversive predictions (which influence Pavlovian values) could result in more ‘votes’ for inaction in the push scenario (Figure ID) than in the trapdoor scenario (Figure IC), leading to a higher proportion of deontological judgments in the former than in the latter. Simultaneously, the model-free system evaluates contextualized actions, assigning negative values to state-action pairs with negative reinforcement histories (e.g., push person 9Cushman F. et al.Simulating murder: the aversion to harmful action.Emotion. 2012; 12: 2-7Crossref PubMed Scopus (180) Google Scholar, 10Blair R. A cognitive developmental approach to morality: investigating the psychopath.Cognition. 1995; 57: 1-29Crossref PubMed Scopus (1041) Google Scholar). An important question is how the model-free system could evaluate actions that have never been performed directly (e.g., violent acts). One possibility is that action values are learned via observation: a recent study showed that observational model-free learning engages similar neural structures as does experiential model-free learning 11Liljeholm M. et al.Dissociable brain systems mediate vicarious learning of stimulus–response and action–outcome contingencies.J. Neurosci. 2012; 32: 9878-9886Crossref PubMed Scopus (22) Google Scholar. Alternatively, the model-based system could train the model-free system through off-line simulations 6Dayan P. How to set the switches on this thing.Curr. Opin. Neurobiol. 2012; 22: 1068-1074Crossref PubMed Scopus (66) Google Scholar. Finally, the Pavlovian system may respond to model-based predictions of aversive outcomes (derived from the scenario text and represented in the decision tree) or, to the model-free aversive values assigned to the actions 4Huys Q.J.M. et al.Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding.PLoS Comput. Biol. 2011; 7: e1002028Crossref PubMed Scopus (224) Google Scholar, 6Dayan P. How to set the switches on this thing.Curr. Opin. Neurobiol. 2012; 22: 1068-1074Crossref PubMed Scopus (66) Google Scholar. system ‘votes’ for its preferred and choices are a of their combined can explain the in judgments for the push and trapdoor cases by that a person and a switch in of both their reinforcement histories and their outcomes, which in influence the ‘votes’ of the model-free and Pavlovian systems (Box 1). Consider a of moral people distinguish between harm performed as a to a and harm as a (Box This can be by the trapdoor with the a trolley is hurtling out of control down the tracks toward five workers, who will die if you do nothing. You can flip a switch that will the trolley onto a different set of tracks, where a large man is Is it morally permissible to flip the switch, killing the large man but saving the five workers? the that outcomes are in these people the switch in the trapdoor to be than in the the and searching through a decision tree can be conceptualized as a set of internal actions that may be to Pavlovian biases 6Dayan P. How to set the switches on this thing.Curr. Opin. Neurobiol. 2012; 22: 1068-1074Crossref PubMed Scopus (66) Google Scholar. One is the of of that lead to aversive or a pruning of the decision tree 7Huys Q.J.M. et al.Bonsai trees in your head: how the Pavlovian system sculpts goal-directed choices by pruning decision trees.PLoS Comput. Biol. 2012; 8: e1002410Crossref PubMed Scopus (228) Google Scholar. pruning is Pavlovian in that it is reflexively by aversive and even when the of rewards aversive 7Huys Q.J.M. et al.Bonsai trees in your head: how the Pavlovian system sculpts goal-directed choices by pruning decision trees.PLoS Comput. Biol. 2012; 8: e1002410Crossref PubMed Scopus (228) Google between and cases is the of harm within the decision tree, which has consequences for action in the of Consider that model-based action the outcomes from all of the decision tree, and Pavlovian pruning of aversive outcomes in a of all the outcomes in a In pruning in a higher value of by the of the aversive outcome of killing one the and in The is to saving the five it can be the of saving five toward the action This of for negative is not possible in Here, the aversive outcome of killing one is to the outcome of saving pruning the aversive outcome also the in although pruning the of the negative outcome of killing one as in it also the of the outcome of saving five the and in The divergence in judgments between and cases may be by the possibility that in the of the outcome of saving five less to the action value in cases which it is to cases which it is and searching through a decision tree can be conceptualized as a set of internal actions that may be to Pavlovian biases [6Dayan P. How to set the switches on this thing.Curr. Opin. Neurobiol. 2012; 22: 1068-1074Crossref PubMed Scopus (66) Google Scholar. One is the of of that lead to aversive or a pruning of the decision tree 7Huys Q.J.M. et al.Bonsai trees in your head: how the Pavlovian system sculpts goal-directed choices by pruning decision trees.PLoS Comput. Biol. 2012; 8: e1002410Crossref PubMed Scopus (228) Google Scholar. pruning is Pavlovian in that it is reflexively by aversive and even when the of rewards aversive 7Huys Q.J.M. et al.Bonsai trees in your head: how the Pavlovian system sculpts goal-directed choices by pruning decision trees.PLoS Comput. Biol. 2012; 8: e1002410Crossref PubMed Scopus (228) Google Scholar. A between and cases is the of harm within the decision tree, which has consequences for action in the of Consider that model-based action the outcomes from all of the decision tree, and Pavlovian pruning of aversive outcomes in a of all the outcomes in a In pruning in a higher value of by the of the aversive outcome of killing one the and in The is to saving the five it can be the of saving five toward the action This of for negative is not possible in Here, the aversive outcome of killing one is to the outcome of saving pruning the aversive outcome also the in although pruning the of the negative outcome of killing one as in it also the of the outcome of saving five the and in The divergence in judgments between and cases may be by the possibility that in the of the outcome of saving five less to the action value in cases which it is to cases which it is In the trapdoor the large man is as a his body is instrumental in the it from the five In the the of the large man is a of the action performed to save the five The of the decision tree is the may arise from Pavlovian pruning of model-based tree search (Box 7Huys Q.J.M. et al.Bonsai trees in your head: how the Pavlovian system sculpts goal-directed choices by pruning decision trees.PLoS Comput. Biol. 2012; 8: e1002410Crossref PubMed Scopus (228) Google Scholar. Because the of model-based, model-free, and Pavlovian systems is the current offers a of For example, the to a in model-free and model-based evaluations 3Balleine B.W. O’Doherty J.P. Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action.Neuropsychopharmacology. 2009; 35: 48-69Crossref Scopus (1182) Google Scholar, 5Wunderlich K. et al.Mapping value based planning and extensively trained choice in the human brain.Nat. Neurosci. 2012; 15: 786-791Crossref PubMed Scopus (218) Google Scholar. The that model-free evaluations to deontological with the possibility that the model-free values into moral can account for two findings in the that physical contact cases such as with deontological the 1Greene J.D. The cognitive neuroscience of moral judgment.in: Gazzaniga M. The Cognitive Neurosciences. 4th edn. MIT Press, 2009: 987-999Google and that with a toward deontological judgments in those cases M. et to the moral 2007; PubMed Scopus Google Scholar. Pavlovian aversive predictions have been to function 4Huys Q.J.M. et al.Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding.PLoS Comput. Biol. 2011; 7: e1002028Crossref PubMed Scopus (224) Google Scholar, 7Huys Q.J.M. et al.Bonsai trees in your head: how the Pavlovian system sculpts goal-directed choices by pruning decision trees.PLoS Comput. Biol. 2012; 8: e1002410Crossref PubMed Scopus (228) Google if such predictions a in deontological function deontological which has been et influences moral and through on harm Sci. PubMed Scopus Google Scholar. Finally, is evidence that control from model-based to model-free systems and from to Cogn. Sci. Full Text Full Text PDF Scopus Google Scholar]. account suggests that deontological as et moral decision 2012; Full Text Full Text PDF PubMed Scopus Google Scholar]. One of decision systems is that each provides in certain Model-based control is for simple However, when the decision tree is and tree search computationally model-free and Pavlovian mechanisms systems for similar Consequentialism provides a for good and but in the of or outcomes that are encountered in moral dilemmas, deontological to out of normative of decision-making can normative a This work by the I and for and