Estimating net survival in population‐based cancer studies

This is the authors' abstract. We don't add key points for this paper.

Roche et al.1 conclude that “great errors” occur in “all classical methods” of estimating net survival and suggest that “cancer registries should abandon all classical methods and adopt the new Pohar Perme [PP] method.” The PP method may well be technically superior, but we believe Roche et al. misrepresent the bias in the preferred classical methods in a manner that may unduly cause alarm. It might be some time before the PP method is implemented in the software used by many registries and it would be a great pity if the scientific community dismissed methodologically sound and important research because of Roche and coworker's claims of “great errors.” It is no surprise that the unstandardized “classical methods” reported by Roche et al. are biased.2-5 However, most cancer registries report age-specific or age-standardized estimates of survival. The differences between the age-standardized PP and the corresponding Ederer II estimates (the preferred classical estimator2) were small (< 0.5 and < 1 percent units at 5 and 10 years respectively). Roche et al. compare the PP method to four “classical methods” including one (Ederer I) that has not been widely used for 50 years, another (Hakulinen) that the original author no longer recommends,2 and one (UVM) that shows greater discrepancy than usually observed (see our later comments). They apply the five methods to several cancer sites and selectively cite, e.g., in the abstract, the largest observed differences (ignoring random error and multiple testing) to support their claims that “great errors” occur in “all classical methods.” We do not dispute that the PP method is the only unbiased estimator of net survival. Our criticism is that the comparisons by Roche et al. are not objective; they grossly overstate the magnitude of the bias in the Ederer II method in a manner that could mislead and alarm the research community. An important issue with the PP estimator is the increased variability and lack of stability for long-term survival, particularly for older age groups. This is illustrated in Figure 1 where the PP estimates are shown together with estimates from a univariable model. Our univariable model is a flexible parametric model6 and is conceptually similar but technically different to that used by Roche et al.; we fitted the model separately to each age group and used restricted cubic splines to model the effect of time with no additional covariates. The model-based approach provides greater stability and less variability, i.e., narrower confidence intervals, for long-term estimates. With respect to Figure 1 there are two statements that are undeniably true—the PP method provides unbiased estimates of the true net survival and the true net survival is unknown. The bias in the “classical methods” results from a dependence between cancer mortality and non-cancer mortality. All approaches are unbiased when applied to patients with the same expected survival, e.g., of the same sex, age and year of diagnosis. The bias in the classical methods is smaller when they are applied within narrow age groups, as is done when age standardizing. The bias is largest in the oldest age group, where expected survival is most heterogeneous, but this bias can be reduced by adjusting for age. Unbiasedness is undoubtedly a desirable property in an estimator, but does not mean that all estimates will equal the true value. Estimates are subject to random variation and it so happens that the PP estimator has larger variability than other methods.3 The approach used by Roche et al. to calculate the “bias with the classical methods” is fundamentally flawed. They calculate the difference between the PP estimate and the classical estimate (both estimated with error) and call this difference the “bias” or “error.” Bias is defined as the difference between the expected value of an estimator and the true value, not the difference between two estimates. Roche et al. mention this in the discussion, but appear to ignore the implications when reporting results and making very strong recommendations. Although we cannot guarantee the model-based estimates in Figure 1 are unbiased, we would argue that any bias is small. The two estimates presented in Figure 1 differ in some instances by more than 15 percent units. These differences do not represent the bias in the model-based estimates as Roche et al. would have us believe; they instead represent the large variability. The large variability in long-term survival, particularly for older age groups, is inherent in the data. The PP estimator weights each observation by the inverse of the expected survival, providing a very elegant approach to estimating the true net survival for a cohort of individuals with heterogeneous expected survival times. Relatively few patients aged 75 years or more at diagnosis survive 10 years or more, and those who do will contribute large weight, potentially causing large fluctuations when these individuals die. The model-based approach has lower variance because it relies on assumptions not required by the PP method. We believe these assumptions are appropriate and the resulting stability and lower variance in estimates of long-term survival are desirable properties. We recognize that we focus heavily on methods for estimating long-term survival for the oldest age group, which is of lesser practical interest given the paucity of available data. Our focus on this area is a response to Roche et al.'s choice to cite, in their abstract, results from long-term estimates in a way which we believe misrepresents the truth. The traditional method of age standardization involves first estimating net survival for each of several age groups and then averaging these estimates using weights from the standard population. If age-specific estimates are obtained using the PP method there may be large variability for longer-term survival for older age groups (Fig. 1). These older age groups have relatively large weight in many standard populations (e.g., 58% over the age of 657), which leads to age standardized estimates with large variance (Fig. 1). We are surprised by the large “bias” in the univariable model reported by Roche et al. We have never observed differences in age-specific estimates estimators of the magnitude they report. The model we use has restrictions to ensure that it is not overly influenced by sparse data in the tails. Roche et al. claim to “see no reason to favor any classically used method such as Ederer I, Hakulinen, Ederer II.” One additional reason is that the PP estimator is not available in software commonly used by cancer registries, such as SEER*Stat. We do not believe that immediately abandoning existing software is motivated given the magnitude of the bias for the recommended “classical method” (Ederer II) is not large for many of the analyses most commonly performed by cancer registries. We are pleased to have incorporated the PP estimator into our own software8 and hope the developers of SEER*Stat incorporate it into their software so it may be widely used. We feel, however, that the best of the classical methods are not as inferior as Roche et al. would have us believe. Researchers should also be aware that the lack of bias in the PP estimator comes at a price of higher variance and this becomes an important issue when age-standardization is employed or with small data and/or long follow-up. Paul W. Dickman1Paul C. Lambert1,2Enzo Coviello3Mark J. Rutherford2 1Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden 2Department of Health Sciences, Centre for Biostatistics and Genetic Epidemiology, University of Leicester, United Kingdom 3Epidemiology Unit, Cancer Registry ASL BT,Barletta, Italy