Evaluating the predictive accuracy of ion-channel models using data from multiple experimental designs

Figures

• 

  Figure 1.. A diagram of a patch-clamp experiment performed on a high-throughput, automated patch-clamp platform. A seal is formed between the plate and the cell. Automatically applied pressure is then used to puncture the cell membrane such that an electrical current flows from the inside of the cell, through the membrane, to the amplifier where it is recorded. Each of the squares on the wellplate (left) represents a well in which the current from a single cell is recorded. Figure modified from [15].

• 

  Figure 2.. a-d indicate the four model structures used. Transition rates are parameterised by two parameters: all rates with even indices have rates of the form, k2i=Aexp⁡{−bVm} and all odd numbered rates are of the form, k2i+1=Aexp⁡{bVm}, except kf and kb in the C1 to C2 transition in the Wang model which are both constant rates [30].

• 

  Figure 3.. The protocols used in our experiment, shown in the order that they are applied. Common features present at the start and end of each protocol are used for postprocessing. All protocols are repeated exactly once except d1, Lei et al. staircase protocol. All protocols are used for model validation except d6, which is used only for validation.

• 

  Figure 4.. The order in which the protocols were performed. First two sweeps of d1 (the staircase protocol) are performed, then a single sweep of each of the other protocols are performed before two final sweeps of d1. Then, after the addition of 1μM dofetilide (which should provide a specific IKr block), this sequence of protocols is repeated once more. This allows the subtraction of post-drug traces from pre-drug traces, which mitigates the presence of endogenous (non-IKr) currents and any other artefacts.

• 

  Figure 5.. The Beattie model is fitted to time-series data (taken from Well B20 using protocol d1) using 30 repeated runs of a stochastic optimization method (CMA-ES) from randomized starting points. (a) The protocol used for data collection (d1, the staircase protocol). (b) The occupancy of each state of the Beattie model over the course of the protocol according to the fitted parameter set. (c) The current recorded under this protocol (red) and best model fit (blue). (d) The values found for two transition-rate parameters, p1 and p2, from multiple optimization runs. The blue markers shown in the inset correspond to those results where the RMSE is at most 101% of the minimum value found. (e) The RMSE error for each parameter set. (f) A cross-section through the likelihood surface, starting at our best estimate of the parameters (λ=0, yellow square) and finishing (when λ=1) at a parameter set with identical maximal conductance (g), but where the transition-rate parameters are taken from the model's original publication (Cell 5) [11].

• 

  Figure 6.. The dependence of model predictions on the particular model structure and wells used for model fitting. All panels on the left relate to sweep 1 of the staircase protocol, d1(1): (a), d1 protocol; (b), fits obtained using each of our models for one particular well, C12; and (c), fits obtained using the Beattie Model for each well. Similarly, all panels on the right relate to our validation protocol d6: (d), the d6 protocol; (e), predictions made using parameter estimates obtained from the fits shown in (b), using each model structure, for the same well, C12; and (f) Beattie model predictions of d6 using parameter sets obtained from multiple wells (fits shown in (c)). The model output shown in (c) and (f) is normalized such that the RMSE of each trace is 1.

• 

  Figure 7.. Residuals obtained when fitting the Beattie model to each training protocol (averaged across wells). The values plotted for 1σ^(yi−zi) have been clipped to lie between −100 and +100. Protocol d6, which is used only for validation and not fitting, is shown in grey.

• 

  Figure 8.. The average behaviour of the Beattie model when producing predictions for unseen protocols, averaged across wells. Sections of the protocols highlighted in red show where the model consistently overestimates IKr, as quantified by T equation (5.4). Whereas, sections highlighted in blue show consistent underestimation. The T values are clipped between −100 and +100.

• 

  Figure 9.. A cross-validation heatmap showing the predictive accuracy of the Beattie Model. (a) and (b) show a comparison between the data (red) and the current fitted using the Beattie model from Well D09 (corresponding to the highlighted square in (e). Similarly, (c) and (d) show the d8 protocol and the worst Beattie Model prediction from Well B20 (corresponding to the highlighted square in (f). (e) and (f) show cross-validation heatmaps for Wells D09 and B20, respectively, which are the wells with the lowest and highest average NRMSE across all pairs of fitting and validation protocols.

• 

  Figure 10.. A comparison of the predictive performance of our chosen model structures. Each heatmap shows the average normalized RMSE when the given model is fitted and validated each pair of protocols. Here, the diagonals show the NRMSE obtained during model fitting (Efit), which, in every case, is noticeably lower than the error in any of the corresponding predictions (Epredict).

• 

  Figure 11.. A linear statistical regression model including well- and protocol-dependent effects, Mw,d, largely fits and recapitulates the well- and protocol-dependence in parameter estimates for the Beattie model. Each panel shows the same two parameter estimates obtained from a given well, and highlights the parameter estimates obtained from the staircase protocol (red crosses). The parameter estimates obtained from the same well, but from a different protocol, are shown by the grey circles in each panel. The blue square shows the well-dependent effect according to Mw, our linear model with no protocol-dependent effects (that is, only well-dependent effects), and the blue star shows the sum of the well and protocol effects (for the given well and the d1staircase protocol) according to the Mw,d model. Note that p1 (the abscissae) has been log-scaled, as it is in our linear model. Note that only two of the eight model kinetics parameters (p1 and p2) are shown as an illustration, but the linear model is constructed for all eight.