Skin Sensitization: Challenging the Conventional Thinking—A Case Against 2 Out of 3 as Integrated Testing Strategy

Recent years have seen a surge in development of alternative methods for assessment of skin sensitizers. This has been in order to meet the regulatory and industrial demand for accurate safety assessment without the use of animal experimentation, and to address societal expectations. Today, 3 such assays are formally validated and have achieved OECD Guideline status (EC, 2013, 2014, 2015). However, partly due to insufficient predictive capacity, none of the validated assays have received a recommendation for use as a stand-alone method. In addition, there is a view that accurate hazard identification must be based upon an assessment of several key events in the relevant Adverse Outcome Pathway (AOP). For these reasons it is proposed that tests are used in combinations, forming so-called Integrated Testing Strategies (ITSs), to address deficiencies in the performance of individual methods and to incorporate end points that reflect several key events in the AOP (Hartung etal., 2013; Jaworska and Hoffmann, 2010; Rovida etal., 2015). Although such ITSs may be configured in a number of ways, one heavily advocated approach relies on the majority vote of individual assays, often referred to as a 2 out of 3 ITS. That is, the identification of hazard is dependent upon the chemical eliciting a positive response in at least 2 of 3 test methods. This approach is based on the view that this will provide an increased weight of evidence and thereby greater confidence in hazard characterization and risk assessment. Here, we present arguments opposing the claimed benefits of 2 out of 3 ITSs by examining the theoretical probability propagation of combined outcomes, simulated predictions generated by hypothetical assays, and actual data obtained from recently published literature. We argue that the added value of combining the top-performing assay with other assays with less predictive accuracy is overestimated, if present, and may actually be detrimental. Where xi∈Positive, Negative, for i=1,2,3⁠. Where c and w denotes correct and wrong predictions respectively, and PXi=c and PXi=w are shorter representations for PXi=xTrue Class=x and PXi≠xTrue Class=x where x∈{Positive, Negative}⁠. Using Equation (4) above, the expected theoretical predictive performance of any 2 out of 3 ITS can be evaluated if estimations of the predictive accuracies of the ITS constituents are available. It is easily demonstrated that if 3 equally good assay constituents are used, there is indeed an added value in combining the outputs into a 2 out of 3 ITS. Assuming an accuracy for each assay of 70%, the expected accuracy of an ITS would be 78.4%. However, if the predictive value of one of the assays is increased to an accuracy of 80%, the relative impact and added value of the remaining 2 assays is lowered, as the expected ITS accuracy is now calculated to be 82.6%. Taking this reasoning even further, if the accuracy of the first assay is increased to 90%, the expected accuracy of the ITS is calculated to 86.8%. Here, the ITS actually performs worse than the top performing assay constituent, and the consequence of combining this assay with less predictive assays is in fact unfavorable to the overall test accuracy. For the sake of completeness, it is of obvious interest to observe the expected ITS performance as a function of all possible combinations of ITS constituent performances. Such an exercise was carried out, with results demonstrated in Figure 1. Assuming a fixed predictive accuracy of the first assay at 90%, the ITS predictive accuracy is plotted over 2 varying complementary assays, as calculated by Equation (4). From Figure 1 it is clear that, assuming a high performing assay is available, the requirements of the complementary assay accuracies are quite stringent in order to achieve a substantial added value. Theoretical ITS performance as a function of the accuracies of 3 ITS assay constituents. The first assay performance is fixed at an accuracy of 90%. The predictive accuracy of 2 complementary assays vary along the x- and y-axis, and the ITS performance is represented by the graph contours. Although the situations described above are based on theoretical estimations of expected ITS performances, it is of interest to see how these estimations compare with real data. In the case of chemical sensitization assessment, such data are readily available. Here, we will be studying 2 different datasets from recent literature. The first dataset, published by Natsch etal. (2013), includes predictions of 145 chemicals using 3 different in vitro assays. The second dataset, published by Urbisch etal. (2015), contains data of 213 chemicals from up to 6 different assays. The authors propose a 2 out of 3 ITS using 3 of these assays, which covers data for 180 chemicals. The details of both of these datasets are summarized in Table 1. The predictive performances of each individual assay, along with the performance of proposed 2 out of 3 ITSs, both as observed by the respective authors and as estimated by Equation (4), are presented. Predictive Performances of Published Data Predictive Performances of Published Data The authors of both papers report an increase in accuracy when combining the outputs from 3 different assays, compared with using each assay as a stand-alone method. However, there is a consistent discrepancy between the ITS performance as observed by the respective authors, and what would be expected from the theoretical calculations using Equation (4). Why is that? Recall that equations (1–4) assume that the outcome of each random variable is independent. In reality, this is rarely the case. On the contrary, it is evident from available data that a number of chemicals are inherently hard to accurately predict. This may be due to physical characteristics of the chemical, such as poor solubility or other incompatibility with test systems. False positives are also prevalent among chemical irritants. False predictions may also be due to chemical domain issues, meaning that prediction models have been designed to recognize chemicals of certain molecular structures, but are unsuitable to make predictions on chemicals with vastly different molecular structures. Last, it is evident that weak sensitizers are harder to predict accurately, compared with their stronger counterparts. Taken together, with currently available assays, there is a negative bias in ITSs towards false predictions due to tendencies to misclassify troublesome chemicals with more than 1 assay. This means that the added value of a 2 out of 3 ITS, compared with only using the top performing assay, is actually even smaller compared with what was expected from the theoretical discussion above, and as illustrated in Figure 1. The idea of using the combined output of several assays in 2 out of 3 ITSs has become the most prevailing strategy to account for insufficient predictive performances among currently validated assays for assessment of skin sensitization (Ezendam etal., 2016). Under the right circumstances, the idea is not flawed. Indeed, if a number of assays perform equally well, exhibiting comparable accuracies at moderate levels, the predictive value does increase when combining the outputs, allowing for more accurate safety assessment. However, if a high performing assay is available, the added value of combining it with less accurate assays diminishes and even disappears. In the reality of assessment of chemical sensitizers, the predictive performance of available assays does vary. Thus, we argue that the added value of constructing 2 out of 3 ITSs, compared with only using the top performing assay, is overestimated in the literature, among regulatory bodies, and in the scientific community. Another argument for the value of ITSs has been that accurate safety assessment must rely on a number of assays that each monitor separate key events of the AOP. The reasoning has been that risks of losing valuable information when moving from a systemic response, as achieved by gold standard invivo assays, is mitigated when using a range of in vitro methods, each contributing information from different key events. Again, the idea is not flawed. However, in the case of a 2 out of 3 ITS, information regarding the target key event of a certain assay is in no way passed on to the final prediction model, which consists solely of a majority vote. Thus, the desired positive effect is not achieved. It should, however, be acknowledged that these arguments do not object to the concept of ITSs per se, but are applicable to ITSs based on a majority vote prediction model. More sophisticated ITSs have been proposed, utilizing eg, machine learning algorithms (Luechtefeld etal., 2015; Tsujita-Inoue etal., 2015), or weight of evidence assessment based on Bayesian statistics (Jaworska etal., 2015). However, such proposed ITSs are still very much in the conceptual phases, and are far from regulatory acceptance or widespread use. In addition, many variable inputs such as in silico parameters of chemico-physical properties require an expertise that is becoming more widespread among strategy developers, but is rarely seen among end users. Finally, the economic burden of using multiple in vitro endpoints together with the suggested in silico modeling expertise is substantial. Thus, until feasible options are available, the added value of combining assay endpoints should be carefully evaluated and cost-benefit estimations should be taken into account. In situations where one sufficiently accurate method is available, the option of relying only on the top performing assay should not be strictly dismissed.