The Numbers of Medicine: A Guide to Reading the Literature, Part 2

Introduction Physicians, especially emergency physicians, are often empirically oriented, relying heavily on clinical experience as well as diagnostic testing and the published literature to reach conclusions about patients. In earlier publications, I provided a discussion of some important statistical concepts which may influence physician and provider thinking.1,2 In this article, I will present and discuss some additional concepts, including some bias influences and logical fallacies, which will help the reader to improve critical analysis of medical articles.Likelihood Ratios, Positive and Negative3-10 All testing has calculations derived from how well the test determines, in a given population, the percentage of people with the disease who test positive (true positives), and those who have the disease but test negative (false negatives), as well as those who don't have the disease (true negatives and false positives). The first calculation is called the sensitivity of the test (true positives divided by the sum of the true positives + false negatives), or how many who have the disease have a positive test. The specificity of the test is defined as the number of people in the population who do not have the disease (true negatives) divided by the sum of the true negatives and false positives (those who do not have the disease, but have a positive test), or how many in the population who do not have the disease have a negative test. Most tests, even in a high-risk population, have a high specificity, as many if not most in the studied population will not have the disease. A screening test should be highly sensitive (as you want to target as many as possible with the disease), while a confirmatory test should be highly specific (as you want to exclude as many false positives as possible). Likelihood ratios give one some assurance that a positive test actually represents disease, and a negative one excludes it. The formulae for these are as follows: Positive likelihood ratio is the sensitivity of the test divided by the number of false positives; a number >10 is considered significant. A negative likelihood ratio is the false negative rate divided by the specificity. A value <0.1 is considered significant. Disease-Oriented Effects, Patient-Oriented Effects3,6,9 Assume some new treatment for hypertension lowers the systolic blood pressure by 5 mm, on average, in a high-risk population. However, after 1 year, the endpoints in the study (which might include heart attacks, stroke, death from vascular causes, elevation in serum creatinine, vision loss, new aneurysm, etc.) do not show a statistically significant change (or may even show a significant detriment). Thus, this therapy has a disease-oriented effect, lowering the average, or mean, blood pressure, without a patient-oriented effect, reduced incidence of end-point disease, and would not generally be a recommended treatment option. A treatment which actually lowers the clinical risks (endpoints), a patient-oriented outcome, would often be recommended. A treatment may have both a disease-oriented effect and a patient-oriented effect, if there is a clinically significant improvement in a predetermined endpoint for the patient, as well as an overall improvement in the measurements, on average, of the general population. Measurements of Central Tendency1,5,7,8,9 Many of us are generally familiar with the “bell-shaped curve,” which includes results in a large part of, but not necessarily the entire, population. When a lab study gives a range of “normal,” often derived from a large sampling of many patients having the same test, these ranges are usually ±2 standard deviations from the mean, thus including 95% of the population within the range. This is sometimes referred to as a confidence interval. If it included 99% of the population, it would represent ±3 standard deviations from the mean. For example, assume the mean of a particular test or finding in a large population is 10 and the standard deviation is 2. To include the true mean, to a 95% confidence interval, a study would describe it as 10 ± 4, or 6-14; for a 99% confidence interval, it would be reported as 10 ± 6, or 4-16. The mean is the average value in a sample result. Thus, in a sample with data of 3, 5, 6, 8, 12, 15, 18, and 21, the mean is 11 (the sum is 88, divided by the number of results, 8). Mode is the most commonly occurring value (in a sample of 3, 3, 7, 8, 12, 15, 22, 25, 29), 3 is the most commonly occurring term, but it gives little insight into actual central tendency, as it may be skewed; in this sample, the mean would be 13.8 (124 divided by 9). Median refers to the middlemost score in a data set. In the first sample above, the median would be 10 (halfway between 8 and 12, even though this wasn't an actual score; with an even number of data points, you split the difference between the middle 2), and in the 2nd, it would be 12, the 5th, or middlemost, term in a set of 9. The calculation of standard deviation is beyond the scope of this paper; suffice it to say, however, that the results in a statistically large enough population suggest that 34% of results will be 1 standard deviation above (or below) the mean, and 68% will be within 1 standard deviation above and below the mean (assuming a population with a normal distribution of results), 95% will be within 2 standard deviations above and below the mean, and 99% will be within 3 standard deviations of the mean. So, if you're a member of Mensa, you're almost 3 standard deviations above the mean, based on whatever test criteria you passed. I guess the rest of us are in Densa. Precision and Accuracy6,10,11 Precision refers to a test that persistently gives you the same, or similar values, within a confidence level, with repeated testing; the results may, or may not, be accurate. It may also be considered reproducibility. Accuracy refers to a test that gives you the correct result, within a confidence interval. A test may have high or low precision, high or low accuracy, or any other of the 4 possible combinations. Biases, aka System Error12-15 Biases are distortions introduced into a study, whether consciously or not, when the study design is flawed. These can directly impact the conclusions that the researcher draws and influence the meaning that a reader gleans. The list is by no means exhaustive but is intended to focus on some of the most common areas of concern. Publication Bias There are many thousands of articles in the various medical and scientific fields published, on average, every day. These represent a sampling of the research done and submitted for publication, though only a small fraction are actually published in the most rigorous of journals (and there are over 52,000 active journals, though many with a low impact factor of influence on scientific thought). Negative results rarely get published, unless the findings refute long-standing practice. Thus, relying solely on what is published skews one's understanding of the topic in question. After all, if one does 20 identical experiments on a particular population, by statistical inference alone, one will be significant to the P < .05 level, and that is the one that likely would be published. Lead Time Bias A hypothetical new test for a serious disease is developed and would allow the clinician to make a diagnosis 6 months earlier than standard testing. However, the earlier diagnosis doesn't change ultimate survival, or reduce morbidity, so there is really no impact on the overall course of the disease, merely a longer period of apprehension for the patient, and clinician. Thus, while the test does allow earlier diagnosis of the disease, it doesn't affect overall mortality or reduce morbidity. Additionally, if there is no specific effective treatment, it creates a period of anxiety for the patient. An example of this was a test for Huntington's chorea. As there is no advantage of knowing early on, except for the likelihood of considering whether to have children and passing on the gene, many people who might be at risk chose not to be tested. Recall or Reporting Bias This occurs in a retrospective study, in which patients who have some untoward effect or result are more likely to remember something that happened than subjects in a population without a similar untoward effect or result. An example might be a population with potential exposure to a carcinogen. Those who develop a cancer associated with the carcinogen are believed to report prior exposure at a higher rate than others without that diagnosis. Incorporation Bias This occurs when part of the screening test is used to make or confirm the diagnosis. For patients with temporal arteritis, an erythrocyte sedimentation rate is used to increase suspicion of the diagnosis, and the sensitivity of ESR includes those with the disease, enhancing the confirmatory testing. Selection Bias, Verification Bias The population chosen is not random, thereby confounding the results. For example, a treatment is tested on 3 different populations: rural, urban, and suburban. Each has distinct characteristics, so the results of the treatment for 1 may not be generalizable to the other 2. Age, Race, Gender, Socioeconomic, Educational, and Case-Mix Bias These are biases introduced because the populations are inherently not equal. One example would be comparing the incidence of finding a malignancy after a diagnostic mammogram in a population of middle-aged women with a population of younger women, neither with special risk factors; another might be lung cancer screening by CT in a smoker and nonsmoker population. Race and gender differences are easily apparent and have impact on potential results. These occur as flaws in study design, where populations with distinct characteristics are lumped together as if the characteristics are insignificant. Anchoring or Hasty Generalization Bias This occurs when the clinician “anchors” on a diagnosis and ignores or refutes alternative diagnoses. Hasty-generalization bias occurs when too little information is available to make an accurate diagnosis, but one is made anyway and acted upon. Logical Fallacies5,6,8,10 Association is not causation, also Post hoc ergo propter hoc (“After the fact therefore because of the fact”). This occurs when two events, which MAY or MAY NOT have any biologically plausible relationship, are assumed to be related. False dilemma occurs when one assumes that there are only two possible choices (besides, say, life and death) and that these are the only options. Medicine is replete with unusual presentations of common diseases, and rare diseases, which often have uncommon manifestations. The absence of evidence is not the evidence of absence. This is a variant of the appeal to ignorance fallacy, whereby one assumes a finding is true because it hasn't been shown to be false. Summary This paper has expanded on some important statistical concepts in clinical medicine, previously discussed, and includes likelihood ratios, disease- and patient-oriented outcomes, measurements of central tendency, confidence intervals, some potential sources of bias, and some logical fallacies.