Answers are grounded in this paper and cite their sources.
You have accessJournal of UrologyJU Forum1 Jul 2023Guidelines for Reporting Observational Research in Urology: The Importance of Clear Reference to Causality Andrew J. Vickers, Melissa Assel, Rodney L. Dunn, Emily C. Zabor, Michael W. Kattan, Maarten van Smeden, and Darren Dahly Andrew J. VickersAndrew J. Vickers *Correspondence: Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, 485 Lexington Ave, New York, NY 10017 telephone: 646-888-8233; E-mail Address: [email protected] Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, New York More articles by this author , Melissa AsselMelissa Assel Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, New York More articles by this author , Rodney L. DunnRodney L. Dunn Department of Urology, University of Michigan, Ann Arbor, Michigan More articles by this author , Emily C. ZaborEmily C. Zabor Department of Quantitative Health Sciences, Cleveland Clinic, Cleveland, Ohio More articles by this author , Michael W. KattanMichael W. Kattan Department of Quantitative Health Sciences, Cleveland Clinic, Cleveland, Ohio More articles by this author , Maarten van SmedenMaarten van Smeden Julius Center for Health Sciences and Primary Care, Utrecht University Medical Center, Utrecht, The Netherlands More articles by this author , and Darren DahlyDarren Dahly School of Public Health, University College Cork, Cork, Ireland More articles by this author View All Author Informationhttps://doi.org/10.1097/JU.0000000000003479AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookTwitterLinked InEmail As statistical editors and reviewers, we have noticed that observational research papers submitted for publication often share a common set of problems with respect to the issue of causality. Causality can be conceptualized in counterfactual terms: would the outcome be different if we change what we do? For instance, we say that smoking causes cancer, because if we reduce smoking, we reduce cancer; conversely, even though ice cream sales closely track the rate of shark bites, we avoid a causal conclusion because banning ice cream would not make ocean swimming any safer. Here we suggest guidelines for reporting of observational studies where investigators calculate statistical associations between an exposure—such as a drug, a change in surgical technique, or a lifestyle factor like smoking or diet—and an outcome—such as cancer diagnosis or cancer progression. The core issue is that authors all too often avoid any explicit reference to causal mechanisms and causal inference but then still try to draw causal conclusions. We repeatedly see investigators declare somberly in the methods section that a study aims merely to “derive statistical associations” but then report results or make recommendations that clearly imply causality: a drug is said to “reduce risk,” a surgical technique to “improve outcome,” or a recommendation made that patients be counseled to change lifestyle so as to “prevent cancer.” Then in response to criticism, authors sometimes deny in their rebuttal letters (and elsewhere) that their language implies a causal conclusion. Assessing statistical association is easy and determining causation is hard; yet while associations are sometimes of research interest, understanding causality is necessary to provide solutions that can improve patient outcomes. Our anecdotal experience has recently been corroborated by systematic reviews of the language used by authors of observational studies: “one section might be carefully phrased in terms of association while the other presented causal language”; “some authors [inappropriately] jumped to recommending acting on the findings” despite reporting only statistical associations.1,2 Of particular note, remarkably few papers explicitly used the word “cause.” One well-known commentary argues that avoidance of causal language has simply become a reflexive response to the difficulties of causal inference, though, naturally, avoiding a subject does not aid scientific progress.3 We will use a hypothetical example to avoid singling out individual authors for what is such a common problem, although the example is closely modeled on a published paper from outside the field of urology. Imagine a study on statins and rates of overdiagnosis following PSA screening. A typical report would introduce the hypothesis in terms of whether “the use of statins is associated with rates of prostate cancer overdiagnosis.” The authors might report lower rates of overdiagnosis in statin users and then rapidly switch to causal language: “Statin use shows protective effects on overdiagnosis. Low-grade cancers in particular were reduced with the use of statins.” A caveat is typically then buried deep in the discussion section (“we cannot fully establish a causal relationship”) and the conclusion is generally vague (a “decrease in overdiagnosis with the use of statins”), something which protects the authors from criticism while allowing them to imply that their paper has causal implications. There is no attempt to identify causal pathways, let alone conduct analyses to explore those pathways. Causal pathways are critical because they determine the implications of the findings for clinical practice. If statins reduce overdiagnosis directly, for instance, through an antineoplastic mechanism, we might more strongly recommend statins to men who are undergoing PSA screening. The clinical consequences would be very different if the effects of statins are indirect. For instance, if statins reduce overdiagnosis by lowering PSA below biopsy thresholds, then the issue for clinical practice is not statin use but the appropriate level of PSA that leads to biopsy. Similarly, we might consider the association between statins and obesity, and between obesity and overdiagnosis, suggesting confounding as an explanation for the study finding. If so, we would ignore statin use altogether and make screening decisions depending on obesity. Once we have thought through causal pathways, we can design appropriate statistical analyses. In our statin example, it is plausible that the apparent effects of statins might in fact be due to PSA hemodilution related to obesity. In brief, men on statins are more likely to be obese, obese men have greater blood volume, and so a given amount of PSA released from the prostate in nanograms will be divided by a greater number of milliliters; this leads to a lower PSA in ng/mL in obese men and thus reduces the chance of biopsy for elevated PSA. To address this in a causal inference framework, we could include obesity as a covariate in a multivariable model testing the association between statin use and overdiagnosis, with an interaction term between statin use and PSA levels. Inclusion of obesity as a covariate addresses whether obesity is confounding the association between statin use and overdiagnosis, whereas inclusion of the interaction term would address whether statin use is only associated with a reduction in overdiagnosis among those with lower PSA levels. Alternatively, it could be that statins could reduce overdiagnosis by having a direct effect on PSA levels. To test this hypothesis, we could compare PSA levels based on statin use and then model the effect of statin use on biopsy rates after adjustment for PSA, perhaps with an interaction term to see if the effect of statin use differs based on PSA level. The exact details and methods of these analyses are not critical here: the key point is to illustrate how causal mechanisms are identified and then analyses developed accordingly. Methods for conducting analyses investigating causality have, unsurprisingly, received detailed attention in the methodological literature.4,5 Indeed, epidemiologists have spent decades thinking about causality and developing methodologies to establish causal inference, from the well-known Bradford Hill criteria6 to the Rubin potential outcomes framework7 to directed acyclic graphs,8 a method of visualizing causal pathways so as to help determine the appropriate statistical analysis. The epidemiological literature also includes critiques of methods for causal inference, for instance, urging more cautious application of propensity scores9 or advising against the use of E values, a statistic purported to give an estimate of confounding.10 In light of these considerations, we are introducing a set of guidelines for papers published in our urology journals. These guidelines are for observational studies where there is any reasonable question about causality. The Table gives specific examples of language to use and avoid. Authors should be judicious and explicit in their use of causal language. This refers not only to use of the words “cause” or “causal,” but to words that imply causality—such as “effect,” “reduce,” “increase,” or “impact”—or recommendations that depend on a causal claim, such as that “patients should avoid,” “doctors should use,” or “efforts should be made to increase.” This is absolutely not a recommendation to avoid causal language, but to be explicit about causality.3 That said, a term that should probably be avoided is “risk factor,” as this has an uncertain meaning.11 Causality should be discussed in the context of practical action. We think about causes of health states because we want to intervene to improve health. For instance, we know that smoking causes lung cancer (among other diseases) and so we advise patients to quit smoking. Coffee is associated with lung cancer, but this association is mostly likely fully explained by confounding by smoking12 rather than a direct causal effect, and hence we do not make practical attempts to limit coffee consumption to prevent cancer. Thinking in terms of practical action can sharpen causal thinking. For instance, if the causal mechanism for the association between obesity and aggressive prostate cancer is hemodilution, that would lead to very different practical recommendations than if the causal mechanism is alterations in testosterone metabolism.13 There should be explicit reference to causality in the “Introduction” section. This should be related directly to the study question. It is, of course, appropriate in some cases not to have a causal hypothesis, but this should be made explicit and a rationale given. For instance, in a study of smoking and surgery outcomes, the authors might state: “We will not investigate the causal pathways between smoking and postoperative recovery as our purpose is to allow clinicians to counsel patients about their expected postoperative course.” Characterizing a study as descriptive or predictive should have a good rationale, and should not be used just as a shortcut to avoid grappling with causal inference. Where the purpose of the study involves exploration of causal mechanisms: a. Describe possible causal pathways in the “Methods” section. Although this can be done formally, for instance, using directed acyclic graphs, it is also reasonable to describe causal pathways using ordinary language in the main text. In many cases this section can be relatively brief: take, for instance, an observational study comparing 2 different treatment modalities. In other cases, authors will need to describe pathways in more detail, carefully describing mediators, confounders, and colliders. b. Describe statistical methods to address causality. This will often focus on confounding. Authors should go beyond a brief reference to “adjusting” in a multivariable model and should describe the rationale for their choice of specific covariates in the context of causal pathways. These need not be lengthy. Take, for instance, “In our study comparing different treatments for early-stage prostate cancer, it is plausible that disease aggressiveness might affect choice of treatment. Accordingly, we included stage, grade, and PSA as covariates in the multivariable model.” However, in some cases, discussion of analyses to control for confounding might take several paragraphs, as authors would need to describe relationships between measured covariates and pathways of confounding. For instance, diet is a possible confounder in an epidemiological study of exercise and urological cancer. Brief reference to diet being one of a list of covariates would be insufficient: authors would need to assess how well their measure reflects confounding by diet, particularly with respect to when diet was measured relative to the likely time course of carcinogenesis. Where the causal question is one of mediation, a mediation analysis should be considered.14 Authors should also note that confounding is not the only threat to causal inference, and other causal criteria should be explored. One criterion is that exposed and unexposed participants should be represented across the distribution of confounders (the “positivity” assumption). For instance, it would be hard to draw conclusions about the effects of chemotherapy if all (or almost all) chemotherapy patients were stage 3 or 4, whereas the patients not receiving chemotherapy were all stage 1 or 2. Another criterion is that exposed participants have similar levels (or dose) of exposure—this is why studies on smoking look at pack-years rather than smoking status—and also that the exposure of 1 participant does not affect the outcome of an unexposed participant (the reason why studies of vaccine effectiveness are often done at the community level). This criterion is known as the “stable unit treatment value assumption” or “SUTVA.” c. Assess control of confounding in the “Results” section. As discussed in the guidelines for reporting statistics in urology,15 authors should avoid assuming that once a multivariable or a propensity approach has been implemented, confounding is no longer a concern. One good rule of thumb is that “differences in measured confounders imply differences in unmeasured confounders,” where the latter includes both measurement error in covariates (sometimes called “residual” confounding) and variables not included in the analysis. For instance, in a study with a survival end point, researchers will not record all comorbidities (such as a rare genetic disorder); moreover, a measured comorbidity (say, diabetes) will often be recorded as present or absent, even though the severity of diabetes can vary between patients. Accordingly, a key analytic step is to compare confounders between exposure groups, for instance, in a “table 1.” If, for instance, the prevalence of diabetes is higher in 1 group than another, it is also likely that the severity of diabetes will be higher in that group. As a result, including diabetes in a multivariable model will not completely control for differences between groups in the case where diabetes severity is associated with survival. Authors might also evaluate how well the measured covariates predict outcome. For instance, the higher the discrimination, the lower the likelihood of unmeasured confounding. That said, we are suspicious of analyses that aim to assess confounding numerically, such as the E value,10 or evaluation of a binary, unmeasured covariate16: the influence of confounding on judgments of causality cannot in our view be reduced to 1 or 2 numbers. d. Carefully assess causal inference and causal estimates in the “Discussion” section. The discussion section should evaluate causal inference in light of the causal pathways described in the “Introduction” and “Methods” sections, the findings reported in the “Results” section, and the relevant scientific literature, whether comparable observational studies or basic science research. Control of confounding and other sorts of bias should be a major consideration. Authors should also reflect on the size of the causal estimate. For instance, in 1 study, high coffee consumption was associated with a greater than 50% reduction in the risk of lethal prostate cancer.17 This is far higher than the chemopreventive effects of pharmaceutical agents targeting specific carcinogenic pathways—for instance, tamoxifen and raloxifene reduce the risk of breast cancer by about 40%—and is thus implausible. e. Draw implications for research and/or clinical practice in light of causal findings. Authors should avoid general references to “association” in their conclusion section and instead make recommendations for research or clinical practice. Such recommendations need to be explicit, such as mechanistic investigation of causality as a research implication or use of a treatment as a clinical practice implication. Implications also need to be specific. Authors should avoid vague calls for further research and instead give details of how such research should be conducted given the particular findings of the current study. Table. Examples of Language to Use and Avoid Scenario Language to use Language to avoid Causality not investigated or not demonstrated • “Associated with …” • “Causal mechanisms will require further evaluation” • “Affects”/“Impacts”/“Increases”/“Reduces” • “Surgeons should avoid …” Introduction section of an observational study • “This is a preliminary study to assess the association between statins and risk so as to motivate subsequent research on causal mechanisms, were such an association to be found” • “We are interested in the variants as prognostic markers, and will not be investigating causal pathways here” • “We hypothesized that any association between smoking and prostate cancer death would be causally related either to lower uptake of PSA screening in smokers or to overall poorer access to care in this group” • “The purpose of the study is to determine which surgical approach results in a lower complication rate” (Note: this implies a causal relationship) • “We aimed to assess the effects of neoadjuvant therapy” (Note: this implies a causal relationship) • “We aimed to assess the relationship between…” without discussing whether causality or causal mechanisms will be addressed • “Our aim was to evaluate the association between neoadjuvant therapy and oncologic outcome” (Note: this avoids causality when only the causal question is interesting here) Covariates not well balanced between groups • “As there was a difference in comorbidity between the 2 surgical approaches, we evaluated the association between comorbidity and surgical outcome” • “Although statistically significant, the number of high-grade tumors was only slightly lower in the high-dose group and is unlikely to explain the very large difference in oncologic outcome” • “The baseline risk of patients receiving blood transfusions was so much higher than those who did not receive transfusion that any causal inference is unsound. We recommend that questions of transfusion approaches in this population can only be addressed by randomized trials” • “Case mix differences do not explain the superior outcomes in the surgery arm because stage, grade, and PSA were included in a multivariable model” • “Our propensity score approach simulates a randomized trial meaning that the estimate for the difference between groups is not subject to confounding” • “Our E value is high and therefore unmeasured confounding is unlikely” Discussion section • “There are 2 possible causal mechanisms for the reported association other than an effect of the drug. The first is … which we evaluated by … and found … The second is … which we evaluated by … reporting … Hence, we conclude that a causal effect of the drug is the best explanation of our findings” • “The difference between groups is larger than could be reasonably explained by the alternative surgical approaches, suggesting considerable unmeasured or residual confounding” • “We found an association between X and Y. The causal pathways between X and Y require further elucidation” • “We conclude that statins are associated with lower cancer risk” (Note: conclusions should include recommendations for research or clinical practice, such as that the causal mechanisms should be explored in subsequent studies) In conclusion, the current literature too often dances around causal issues, avoiding direct reference and tiptoeing back and forth between cautious language about association (“correlation does not imply causation”) and incautious language that strongly implies a causal relationship (“patients should avoid X”). We propose these guidelines to ensure that observational research papers clearly reference causality when describing study aims, and, when the aim is causal, to keep causality first and foremost in design, analysis, interpretation, and conclusions. Following these guidelines will help ensure causal conclusions are applied judiciously and communicated clearly so as to help clinicians and patients make better decisions about health. REFERENCES 1. . Consistency of causal claims in observational studies: a review of papers published in a general medical journal. BMJ Open. 2021; 11(5):e043339. Medline, Google Scholar 2. Causal and associational linking language from observational research and health evaluation literature in practice: a systematic language evaluation. medRxiv2021; 10.1101/2021.08.25.21262631v2. Google Scholar 3. . The C-word: scientific euphemisms do not improve causal inference from observational data. Am J Public Health. 2018; 108(5):616-619. Crossref, Medline, Google Scholar 4. . Thinking clearly about correlations and causation: graphical causal models for observational data. Adv Methods Practices Psychol Sci. 2018; 1(1):27-42. Crossref, Google Scholar 5. . Formulating causal questions and principled statistical answers. Stat Med. 2020; 39(30):4922-4948. Crossref, Medline, Google Scholar 6. . The environment and disease: association or causation?. Proc R Soc Med. 1965; Medline, Google Scholar . Causal effects in clinical and epidemiological studies potential and Public Health. Crossref, Medline, Google Scholar . Use of directed acyclic to identify confounders in applied health review and J 2021; Crossref, Medline, Google Scholar . on statistical reporting to help improve of a statistical Am J Med. 2020; Crossref, Medline, Google Scholar . The of in bias analysis. 2021; Crossref, Medline, Google Scholar . a risk factor for being a Crossref, Medline, Google Scholar . Coffee and risk of lung Cancer Crossref, Medline, Google Scholar . of hemodilution of and Crossref, Medline, Google Scholar . effect and 2018; Crossref, Medline, Google Scholar . for reporting of statistics for clinical research in urology. Crossref, Medline, Google Scholar . Assessing to an covariate in an observational study with outcome. J R Stat Soc Google Scholar . Coffee consumption and prostate cancer risk and in the Health J Cancer Crossref, Google Scholar This was in by the of Cancer with a Cancer Center to Memorial Sloan Kettering Cancer Center and a in Cancer to the Center for and of The Authors have no of to This study was from of Urology, and and by Health and All J and to A New to of Urology, of Urology, and and by Health and All Andrew J. Vickers Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, New York *Correspondence: Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, 485 Lexington Ave, New York, NY 10017 telephone: 646-888-8233; E-mail Address: [email protected] More articles by this author Melissa Assel Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, New York More articles by this author Rodney L. Dunn Department of Urology, University of Michigan, Ann Arbor, Michigan More articles by this author Emily C. Zabor Department of Quantitative Health Sciences, Cleveland Clinic, Cleveland, Ohio More articles by this author Michael W. Kattan Department of Quantitative Health Sciences, Cleveland Clinic, Cleveland, Ohio More articles by this author Maarten van Smeden Julius Center for Health Sciences and Primary Care, Utrecht University Medical Center, Utrecht, The Netherlands More articles by this author Darren Dahly School of Public Health, University College Cork, Cork, Ireland More articles by this author All This was in by the of Cancer with a Cancer Center to Memorial Sloan Kettering Cancer Center and a in Cancer to the Center for and of The Authors have no of to This study was from
No takes yet. Share an insight, caveat, or question.
Vickers et al. (2023) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: