Patterns between evidence seeking behaviors, reasoning, and cognitive reflection: A supervised clustering approach

In the face of risk, decision-making can be driven by styles of evidence gathering and information processing. When studying how people navigate the complex landscapes of evidence, researchers face the analytical problem of an exponentially growing number of distinct evidence gathering styles, as the number of pieces of information increases. The existing solution is to chunk information into a manageable number of pre-defined categories. In this work, we propose to meet this analytical challenge with a two-pronged strategy. First, our observational setting offers more fine-grained pieces of evidence but masks the content of evidence behind a query (e.g., ’what does so-and-so say?’) to ensure that people only access what they deem as potentially relevant. Second, applying supervised clustering based on the SHapley Additive exPlanation (SHAP) methodology allows us to relate evidence from gathered patterns to final decision while significantly relaxing theoretical delineations of evidence types that would otherwise make analysis intractable. We argue that this two-pronged strategy approximately integrates the pathway from evidence seeking to information processing and decision-making. We applied this strategy to demonstrate the fruitfulness of bridging work on evidence gathering and information processing. For example, one mental shortcut (heuristic) to arrive at decisions when assessing a causal claim using a 2 × 2 contingency table is the ‘base rate neglect heuristic’ (considering only the treatment group, comparing the number of positive outcomes to the number of negative outcomes). While base rate neglect is a well-established heuristic in information processing research, there is not yet a clear equivalent picture regarding evidence gathering. We develop this picture by considering the assessment of the effectiveness of a hypothetical nasal spray based on queries that span gathering/processing evidence categories. Using a demographically diverse online sample collected during August 2024 in the United States, we establish that the base rate neglect heuristic from information processing research is also a heuristic when it comes to gathering first-order data. For example, we find that higher performance on the cognitive reflection test predicts selection of the full data of a 2 × 2 contingency table. But, the latter group is nevertheless similar to base rate neglecters in who they consider to be relevant outside sources (i.e., they have roughly the same “deference” behavior). So while evidential categories in information seeking research are well suited to track differences in deference behaviors, they are blind to the difference between these two groups. These findings are additionally important for designing tailored health communication so as to avoid fallacious inferences.