Impact of predictor measurement heterogeneity across settings on the performance of prediction models: A measurement error perspective

Figures

• 

  Figure 1. Measures of predictive performance under predictor measurement error and predictor measurement heterogeneity. The data‐generating mechanism corresponded perfectly to the estimated logistic regression model. The top rows show calibration plots of a single‐predictor model that is fitted using predictor measurement x and validated by re‐estimating the model on the same data using w. The bottom rows show situations where the same model is transported from derivation to validation setting, specifically, the model (D) is derived using x and validated using w, (E) is derived and validated using w, and (F) is derived using w and validated using x. The calibration plots show the calibration slope (black line) and predicted probability frequencies (bottom‐histograms) for situations in which the predictor measurement variance at validation equals 200% (A,D), 100% (B,E), or 50% (C,F) of the predictor measurement variance at derivation. The val.prob function from the rms package was used to compute the simulation outcome measures and to generate the calibration plots,27 where we edited the legend format settings in the plot to improve readability. MV = measurement variance of the predictor…

• 

  Figure 2. Decomposed Brier score under predictor measurement error and predictor measurement heterogeneity. The data‐generating mechanism corresponded perfectly to the estimated logistic regression model. The plot displays the large sample properties of the components of the Brier score (Equation 7) under increasing random predictor measurement variance at validation, corresponding to the random measurement error model (Equation 2). The left panel shows the Brier score for a single‐predictor logistic regression model that is fitted using predictor measurement x and validated by re‐estimating the model on the same data using w. The right panel shows transportation from w at derivation to x at validation (up to %MV = 100) and transportation from x at derivation to w at validation (from %MV = 100 onwards). MV = measurement variance of the predictor measurement used for model validation relative to the predictor measurement used for derivation

• 

  Figure 3. Measures of predictive performance under measurement heterogeneity in one of two predictors in finite sample simulations. Mean c‐statistic, median calibration slope, and mean Brier score averaged over 10 000 repetitions with interquartile range and 95% confidence interval for a two‐predictor model where one of the predictors is measured consistent across settings, whereas the other is measured heterogeneously. Horizontal bars indicate performance measures at model derivation, while boxes indicate performance at external validation. The predictor measurement structure in the derivation set (n = 2000) corresponds to the random measurement error model (Equation 2). In the validation set (n = 2000), predictor measurements consist of varying structures under Equation (1)

• 

  Figure 4. Measures of predictive performance under measurement heterogeneity in both predictors in finite sample simulations. Mean c‐statistic, median calibration slope, and mean Brier score averaged over 10 000 repetitions with interquartile range and 95% confidence interval of a two‐predictor logistic regression model in which both predictors are measured heterogeneously across settings. Horizontal bars indicate performance measures at model derivation, while boxes indicate performance at external validation. Measurements in the derivation set (n = 2000) are recreated using Equation (2), which corresponds to the random measurement error model. In the validation set (n = 2000), measurements correspond to various measurement structures under Equation (1)